Details
-
Bug
-
Resolution: Fixed
-
Major
-
0.7
-
None
-
Unknown
-
Description
Problem
After XWiki is restarted, the chunks indexed since the previous start are missing from the index. Keyword and semantic search return nothing for them, and the chunk list of the collection is incomplete, until Solr has finished replaying its transaction log. Nothing in the UI says that a replay is running.
The data is not lost: the chunks come back when the replay ends. The duration grows with the amount indexed since the previous start. Measured on a local instance with the embedded Solr:
- one restart: a 632 MB transaction log took 52 minutes to replay
- another restart: the replay re-applied 85,665 added chunks and 11,085 delete-by-query operations, and the index showed 1,914 of 65,282 chunks while it ran
Cause
The indexing code never hard-commits the aillm Solr core:
- SolrConnector.storeChunks and the delete-by-query methods send their update with a "commit within" of 10 seconds, then trigger a soft commit.
- In Solr a "commit within" is a soft commit by default, and the core configuration provided by XWiki has no automatic commit.
- A soft commit makes the changes searchable but does not write them to the index on disk, so they exist only in the transaction log.
The only hard commit is the one done when the core is initialized at startup. On XWiki 17.4 a clean shutdown does not commit either.
Expected
Indexed chunks are durably committed, so that a restart finds them in the index without a long replay.
Steps to reproduce
- Index a collection with a few thousand chunks.
- Restart XWiki.
- Search in that collection right after the start: no result until the replay ends (Log replay finished in the log).
Attachments
Issue Links
- relates to
-
LLMAI-115 Improve the indexing speed
-
- Closed
-