Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-234

After a restart, indexed chunks are missing from search until Solr finishes replaying its transaction log

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 1.0.1
    • 0.7
    • None
    • Unknown

    Description

      Problem

      After XWiki is restarted, the chunks indexed since the previous start are missing from the index. Keyword and semantic search return nothing for them, and the chunk list of the collection is incomplete, until Solr has finished replaying its transaction log. Nothing in the UI says that a replay is running.

      The data is not lost: the chunks come back when the replay ends. The duration grows with the amount indexed since the previous start. Measured on a local instance with the embedded Solr:

      • one restart: a 632 MB transaction log took 52 minutes to replay
      • another restart: the replay re-applied 85,665 added chunks and 11,085 delete-by-query operations, and the index showed 1,914 of 65,282 chunks while it ran

      Cause

      The indexing code never hard-commits the aillm Solr core:

      • SolrConnector.storeChunks and the delete-by-query methods send their update with a "commit within" of 10 seconds, then trigger a soft commit.
      • In Solr a "commit within" is a soft commit by default, and the core configuration provided by XWiki has no automatic commit.
      • A soft commit makes the changes searchable but does not write them to the index on disk, so they exist only in the transaction log.

      The only hard commit is the one done when the core is initialized at startup. On XWiki 17.4 a clean shutdown does not commit either.

      Expected

      Indexed chunks are durably committed, so that a restart finds them in the index without a long replay.

      Steps to reproduce

      1. Index a collection with a few thousand chunks.
      2. Restart XWiki.
      3. Search in that collection right after the start: no result until the replay ends (Log replay finished in the log).

      Attachments

        Issue Links

          Activity

            People

              ppantiru Paul Pantiru
              ppantiru Paul Pantiru
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: