Details
-
Improvement
-
Resolution: Fixed
-
Major
-
1.0.0
-
None
-
Unknown
-
Description
Problem
New collections are created with a maximum chunk size of 2000 characters. The default embedding model (all-MiniLM-L6-v2) reads at most 256 tokens, which is roughly 1000 characters of English text. The second half of most chunks is therefore never embedded and cannot be found by semantic search (see LLMAI-173).
Change
The collection template sets the maximum chunk size to 1000 characters, so that a chunk fits in what the default model reads.
Impact
- Only collections created after the upgrade get the new default.
- Existing collections keep their chunk size. To use the new size, change "Chunking max size" on the collection and reindex it.
- A collection indexed with 1000-character chunks has about twice as many chunks as with 2000.
This does not fix LLMAI-173: chunking stays character-based and does not know the limit of the configured model.
Attachments
Issue Links
- relates to
-
LLMAI-173 Chunking is character-based and ignores the embedding model's maximum input length, so chunk tails are silently never embedded
-
- Open
-