Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-175

The Sources list shows everything retrieved, not what the answer actually used

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • None
    • 0.9
    • None
    • XWiki 18.6.0, ai-llm 0.9.
    • Unknown

    Description

      What happens

      RAGChatRequestFilter.extractURLsAndformat walks the entire searchResults list and emits every unique document URL:

      for (Context result : searchResults) {
          String sourceURL = result.url();
          if (!addedUrls.contains(sourceURL)) {
              sourcesBuilder.append(String.format(SOURCE_STRING, sourceURL));
              addedUrls.add(sourceURL);
          }
      }
      

      chatWidget.js then splits that string on newlines and lists every entry. So "Sources" is the raw retrieval set with duplicate URLs collapsed – it has no relationship to which chunks the model actually drew on.

      Why it matters

      With the default budget the user can see up to 10 sources for a question answered entirely from one page. It reads as though the answer synthesised ten documents. That devalues the citation list – users cannot tell a precise retrieval from a noisy one, and cannot tell which link to follow to verify the answer.

      Observed

      A correct answer drawn from a single page was presented with 8 sources, 7 of them unrelated: 5 semantic + 5 keyword = 10 chunks, 1 removed as a duplicate content match, leaving 9 chunks over 8 distinct URLs.

      Suggested fix

      The prompt already numbers the chunks – DEFAULT_CHUNK_TEMPLATE emits a document element with an index attribute. So the cleanest option is to ask the model to cite those indices and render only the cited ones. Cheaper alternatives: show the similarity score next to each source, or list only sources above a relevance threshold, or visually separate "used" from "also retrieved".

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sebels Sebastian Elsner
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: