Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-230

Add a chunking method that follows the document structure and keeps the page hierarchy and headings as context

    XMLWordPrintable

Details

    • New Feature
    • Resolution: Unresolved
    • Major
    • None
    • None
    • None
    • Unknown

    Description

      Chunks are cut by character count and carry no information on where they come from. A text under "Projects > Alpha > Risks" and the same text under "Archive > 2019" get the same embedding, and a chunk often starts in the middle of a section.

      Add a chunking method sections that:

      • keeps sections together and splits large ones at block, sentence and word boundaries;
      • prefixes each chunk with a breadcrumb: the page's position in the wiki hierarchy (ancestors by page name, the page by its title) and the headings above the chunk;
      • works for every syntax with a parser, as it uses the parsed document (depends on LLMAI-176).

      Switching an existing collection to this method takes effect after a manual re-index (without cleaning), which embeds all chunks again.

      Attachments

        Activity

          People

            Unassigned Unassigned
            GOhrner Gunter Ohrner
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: