Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-178

content and content_index store the same analysed text twice

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • Major
    • None
    • 0.9
    • None
    • XWiki 18.6.0, ai-llm 0.9.
    • Unknown

    Description

      What happens

      SolrConnector.getSolrDocument writes the chunk text into two fields:

      solrDocument.addField(AiLLMSolrCoreInitializer.FIELD_CONTENT, chunk.getContent());
      solrDocument.addField(AiLLMSolrCoreInitializer.FIELD_CONTENT_INDEX, chunk.getContent());
      

      Both fields are declared as text_general: content via addTextGeneralField(FIELD_CONTENT, false, false) in createSchema, and content_index via setTextGeneralField(FIELD_CONTENT_INDEX, false, false) in the migration to schema version 121000006. The migration's comment reads:

      // Add another version of the text field, but indexed as regular text.

      which suggests the intent was that content was not analysed as text. On a fresh install it is, so the two fields carry identical analysis of identical text.

      Why it matters

      No correctness impact – purely index size and indexing cost. The chunk text, which is the bulk of the index, is analysed and stored twice. It is also a small maintenance trap: a future reader of createSchema could reasonably conclude the two fields differ in analysis when they do not.

      Suggested fix

      Either drop content_index and search content directly, or make the two fields actually differ as the comment implies – e.g. content stored and not indexed, content_index indexed and not stored. Either way a schema migration and re-index would be needed, so this may be worth folding into the next change that already requires one.

      Noticed while investigating retrieval relevance in 0.9; filing separately as it is cosmetic relative to the other reports.

      Attachments

        Activity

          People

            Unassigned Unassigned
            sebels Sebastian Elsner
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated: