Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-115

Improve the indexing speed

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 0.7
    • 0.6.2
    • None
    • Unknown
    • N/A
    • N/A

    Description

      In particular when, e.g., the actual embedding instantly fails or chunks don't need to be re-embedded, it is noticeable that indexing is quite slow. I found that this is due to two reasons:

      1. The regular expression for finding the last heading at the end of a chunk is surprisingly slow.
      2. The Solr index is committed very frequently (after every document).

      Both can easily be improved.

      Attachments

        Activity

          People

            MichaelHamann Michael Hamann
            MichaelHamann Michael Hamann
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: