Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-235

Lower the default chunk size of new collections to 1000 characters

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 1.0.1
    • 1.0.0
    • None
    • Unknown

    Description

      Problem

      New collections are created with a maximum chunk size of 2000 characters. The default embedding model (all-MiniLM-L6-v2) reads at most 256 tokens, which is roughly 1000 characters of English text. The second half of most chunks is therefore never embedded and cannot be found by semantic search (see LLMAI-173).

      Change

      The collection template sets the maximum chunk size to 1000 characters, so that a chunk fits in what the default model reads.

      Impact

      • Only collections created after the upgrade get the new default.
      • Existing collections keep their chunk size. To use the new size, change "Chunking max size" on the collection and reindex it.
      • A collection indexed with 1000-character chunks has about twice as many chunks as with 2000.

      This does not fix LLMAI-173: chunking stays character-based and does not know the limit of the configured model.

      Attachments

        Issue Links

          Activity

            People

              ppantiru Paul Pantiru
              ppantiru Paul Pantiru
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: