Hybrid search sorts BM25 and cosine scores in one comparator, so keyword hits always outrank semantic hits

What happens

DefaultCollectionManager.hybridSearch concatenates the semantic and the keyword result lists, de-duplicates on identical chunk content, and hands the merged list to filterSearchResults, which re-sorts it:

// Sort the results by similarity score in descending order again as the sorting was lost during grouping.
.sorted(Comparator.comparingDouble(Context::similarityScore).reversed())

The two halves populate Context.similarityScore from different, unrelated scales:

• the semantic half is a Solr kNN query on a DenseVectorField with similarityFunction=cosine. Lucene's VectorSimilarityFunction.COSINE returns (1 + cos) / 2, so the score is bounded to 0..1.
• the keyword half is an ordinary Solr query scored by BM25, which is unbounded and in practice lands between about 1 and 30.

Since the semantic score can never exceed 1.0, every keyword hit that scores above 1.0 – i.e. essentially every real keyword match – is sorted above every semantic hit, no matter how good the vector match was or how weak the keyword match was.

Why it matters

The merged list is both the LLM's context, in order, via RAGChatRequestFilter.buildContext, and the user-visible Sources list. So the strongest semantic hit is pushed below the weakest keyword hit in the prompt and in the citations. Enabling keyword search therefore actively degrades the ordering that vector search got right.

Observed

Query: "How do I install zsh and oh my zsh at rise?", one collection, maxSemanticResults=5, maxKeywordResults=5. The only relevant page in the wiki is Development/zsh, and it is the top hit in both halves independently:

• keyword-only: zsh 21.63, then four chunks that do not contain the string "zsh" at all (5.70 / 5.39 / 5.36 / 5.33)
• semantic-only: zsh 0.818, then unrelated chunks (0.691 / 0.686 / 0.686 / 0.684)
• hybrid: the four irrelevant keyword chunks come first (5.70 ... 5.33) and Development/zsh lands at position 5 of 9.

Suggested fix

Do not compare the two scores directly. Options, roughly in order of robustness:

1. Reciprocal Rank Fusion over the two ranked lists – the standard hybrid-search answer, needs no score comparability at all.
2. Min-max normalise each list to 0..1 before merging.
3. Interleave the two lists round-robin by rank.

Whichever is chosen, Context.similarityScore would benefit from documenting which scale it carries, since REST and MCP consumers see it too.