Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-174

No relevance floor for either retrieval half, and none is configurable

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • None
    • 0.9
    • None
    • XWiki 18.6.0, ai-llm 0.9, bundled Solr 9.4.1.
    • Unknown

    Description

      What happens

      Neither retrieval path applies any minimum score:

      • SolrConnector.similaritySearch issues a Solr kNN query with topK set to the configured limit. A kNN query returns the N nearest neighbours unconditionally, however far away they are.
      • SolrConnector.keywordSearch uses setRows(N).

      RAGChatRequestFilterClass exposes only maxSemanticResults and maxKeywordResults. There is no threshold property anywhere in the application, so this cannot be worked around by configuration.

      Why it matters

      When a question is answerable from a single page, the remaining slots are filled with the best of the irrelevant – noise by construction rather than a matching failure. Those chunks go into the LLM's context and into the user-visible Sources list. Lowering the two limits reduces the noise but also caps genuinely multi-document answers, so it is not a substitute for a floor.

      Observed

      The irrelevant floor is well separated from the relevant hit, so a threshold would work cleanly:

      • query "How do I install zsh and oh my zsh at rise?" – relevant chunk 0.818, unrelated chunks 0.684-0.691
      • query "zsh" – relevant chunk 0.714, unrelated chunks (a staff directory) 0.628-0.665

      A semantic floor anywhere in 0.70-0.75 separates signal from noise on both. These are Lucene cosine scores, i.e. (1 + cos) / 2.

      Suggested fix

      Add optional minSemanticScore / minKeywordScore properties on RAGChatRequestFilterClass, honoured by hybridSearch and exposed on the REST and MCP search APIs. For the semantic half Solr can enforce it directly by wrapping the kNN query in a function-range filter; otherwise post-filtering in collectResults is enough. Defaulting to no floor keeps current behaviour for existing installs.

      Closely related to the score-scale issue filed alongside this one: a single floor cannot be expressed while the two halves report incomparable scores, so the two are best fixed together.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sebels Sebastian Elsner
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: