Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-177

Attachment indexing cannot be disabled, restricted by MIME type, or size-capped

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • None
    • 0.9
    • None
    • XWiki 18.6.0, ai-llm 0.9.
    • Unknown

    Description

      What happens

      XWikiDocumentDocument.getFormattedAttachments() runs every attachment of every indexed document through Tika and appends the extracted text to the content that gets chunked and embedded:

      String content = this.tika.parseToString(attachment.getContentInputStream(context));
      if (StringUtils.isNotBlank(content)) {
          return Stream.of(formatHeading(2, attachment.getFilename()) + content);
      }
      

      There is no MIME allow-list, no size gate, and no configuration option anywhere to turn attachment indexing off. Tika.parseToString applies its default 100k-character write limit, which bounds how much of any single attachment is taken but does not filter which attachments are taken at all.

      Why it matters

      An administrator who points a collection at a space gets that space's attachments indexed whether or not that is wanted. There is no way to exclude binary formats, scanned documents, exports or archives whose extracted text is noise, and no way to see how much of the index attachments account for. Extracted attachment text also competes for the same result slots as page content, and inherits the chunking behaviour of the page it hangs off.

      For large wikis this is also an indexing-cost issue: every attachment is parsed on every re-index.

      Suggested fix

      Add per-collection options: enable or disable attachment indexing, a MIME type allow-list, and a maximum attachment size. Defaulting to current behaviour keeps existing installs unchanged.

      Filed separately from the raw-wiki-syntax issue in the same method, as requested during triage of that report.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sebels Sebastian Elsner
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: