Uploaded image for project: 'XWiki Platform'
  1. XWiki Platform
  2. XWIKI-24846

Make the code macro's syntax highlighter pluggable and use a client-side one by default

    XMLWordPrintable

Details

    • New Feature
    • Resolution: Unresolved
    • Major
    • None
    • 18.7.0
    • Rendering - Code Macro
    • Unknown

    Description

      Context

      This implements the proposal accepted on
      Change the code macro implementation
      (+1 from mleduc, tmortagne, mflorea and Wardenburg), whose conclusion was:

      1) We agreed to move to a javascript-based syntax highlighting in XS
      2) We agreed to keep the code macro and introduce an Admin UI to configure which implementation
      to use and by default to use the javascript-based one. And once done and working well, to move the
      pygments-based one to contrib.

      The reasons recorded there:

      • Performance: the current highlighter is written in Python and executed through Jython. It is very
        slow on first use, while Jython loads, and still slow afterwards, especially on large content.
      • Missing features: line numbering, a button to copy the content (XWIKI-21364), and others, which
        are far easier to provide client side.
      • The historical reason for keeping the highlighting server side – the PDF exporter not supporting
        JavaScript – no longer applies, since the current PDF exporter does support it.

      A client-side implementation already exists in contrib and works:
      Prism Macro, along with
      org.xwiki.contrib.prismjs:prismjs-code which substitutes it for the code macro
      (https://github.com/xwiki-contrib/prismjs/tree/master/prismjs-code). What is left is doing it
      properly in XWiki Standard.

      Current state in the code

      Most of the pluggability is already there:

      • CodeMacro resolves a HighlightParser component by language hint, and otherwise falls back
        to componentManager.getInstance(HighlightParser.class, "default") – the fallback hint is
        hardcoded.
      • PygmentsParser is declared as {{@Component(roles = {HighlightParser.class}

        )}} with no
        @Named, so it is that default hint.

      • HighlightParser#highlight(String syntaxId, Reader source) receives only the language and the
        content: not the macro parameters, and not whether the call is inline.
      • xwiki-platform-rendering-macro-code declares jython-slim and pygments as direct
        dependencies, so the generic macro and the Pygments implementation ship in the same module.
      • The source parameter needs no work: the generic part of the macro already resolves it and hands
        the resulting content to the HighlightParser.

      Proposal

      1. Replace the hardcoded "default" fallback hint by a configured one, and add the Admin UI to
        choose among the HighlightParser implementations present in the instance. Per-language
        HighlightParser components keep taking precedence, as today.
      2. Extract the Pygments implementation into its own module, under the hint pygments, so that
        xwiki-platform-rendering-macro-code stops depending on Jython and Pygments.
      3. Add a client-side HighlightParser implementation to XS, based on Prism.js, and make it the
        configured default.
      4. Once that is stable, move the Pygments module to contrib.

      Also to be decided while doing 1: whether HighlightParser#highlight needs the macro parameters
      and an inline flag, which a client-side implementation may need and the current signature cannot
      carry.

      Backward compatibility

      The risk identified on the thread is language coverage, not architecture:

      • Pygments supports far more languages than the JavaScript frameworks. The two that matter to XWiki
        are Velocity – supported by Pygments only because the project asked for it – and Groovy. Prism.js
        supports both, which is a concrete argument for Prism over highlight.js, which does not.
      • Language identifiers do not map one to one. Pygments treats bash, sh, ksh, zsh and
        shell as synonyms; Prism does not, and zsh would need mapping onto shell. The agreed
        approach is to handle such cases individually inside the Prism implementation rather than build a
        generic mapping layer.

      Existing content using the code macro must keep being highlighted after the switch.

      Consequences for other issues

      • XWIKI-21364 (copy button) and XWIKI-22832 (line numbers) become much cheaper client side.
      • XWIKI-8456 (cache the highlighting result for performance) largely dissolves.
      • XWIKI-17107 (upgrade Pygments) and XWIKI-18369 (Pygments lexer patch) become contrib concerns once
        the Pygments implementation moves out.

      Attachments

        Activity

          People

            Unassigned Unassigned
            Lucas C Charpentier Lucas
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: