Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
None
-
0.9
-
None
-
XWiki 18.6.0, ai-llm 0.9, bundled Solr 9.4.1.
-
Unknown
-
Description
What happens
Neither retrieval path applies any minimum score:
- SolrConnector.similaritySearch issues a Solr kNN query with topK set to the configured limit. A kNN query returns the N nearest neighbours unconditionally, however far away they are.
- SolrConnector.keywordSearch uses setRows(N).
RAGChatRequestFilterClass exposes only maxSemanticResults and maxKeywordResults. There is no threshold property anywhere in the application, so this cannot be worked around by configuration.
Why it matters
When a question is answerable from a single page, the remaining slots are filled with the best of the irrelevant – noise by construction rather than a matching failure. Those chunks go into the LLM's context and into the user-visible Sources list. Lowering the two limits reduces the noise but also caps genuinely multi-document answers, so it is not a substitute for a floor.
Observed
The irrelevant floor is well separated from the relevant hit, so a threshold would work cleanly:
- query "How do I install zsh and oh my zsh at rise?" – relevant chunk 0.818, unrelated chunks 0.684-0.691
- query "zsh" – relevant chunk 0.714, unrelated chunks (a staff directory) 0.628-0.665
A semantic floor anywhere in 0.70-0.75 separates signal from noise on both. These are Lucene cosine scores, i.e. (1 + cos) / 2.
Suggested fix
Add optional minSemanticScore / minKeywordScore properties on RAGChatRequestFilterClass, honoured by hybridSearch and exposed on the REST and MCP search APIs. For the semantic half Solr can enforce it directly by wrapping the kNN query in a function-range filter; otherwise post-filtering in collectResults is enough. Defaulting to no floor keeps current behaviour for existing installs.
Closely related to the score-scale issue filed alongside this one: a single floor cannot be expressed while the two halves report incomparable scores, so the two are best fixed together.
Attachments
Issue Links
- is related to
-
LLMAI-171 Hybrid search sorts BM25 and cosine scores in one comparator, so keyword hits always outrank semantic hits
-
- Open
-
-
LLMAI-172 Keyword search turns a natural-language question into an unweighted OR bag of words with no stopword filtering
-
- Open
-
- relates to
-
LLMAI-175 The Sources list shows everything retrieved, not what the answer actually used
-
- Open
-