The code macro syntax highlighting is slow and scales badly when several pages are rendered in parallel

Pages with many code macros are slow to render, and they get much slower when many of them are rendered at the same time (for example while crawlers visit a wiki): each code macro is highlighted on the server by Pygments running in Jython, and the highlighting has costs that don't depend on the code being highlighted and that limit how well it runs on several threads.

Causes

• PygmentsParser passes the Python script that drives Pygments as a string to the script engine for each highlight, so Jython parses and compiles it again every time and loads a new Java class for each highlight.
• Pygments lexers keep their token rules in Python lists shared by all threads, and Jython synchronizes each step of the iteration over a list. The lexer iterates over these rules at each position of the code, so the threads highlighting code at the same time block each other.
• The text of each token is parsed with the plain text parser, which allocates a 16 KB read buffer for each call. This one is fixed in the plain text parser, see the linked XRENDERING issue.

Measurements

Highlights per second of a 25-line Java snippet on an 8-core laptop, measured with PygmentsParserScalingTest. The control is a pure-Java regular expression tokenizer without shared state, which shows how far the machine itself scales.

With both changes, the highlighting is 2.2 times faster on 1 thread and 2.1 times faster on 8 threads, and it scales like the control. With only the changes of this issue, the allocations of the plain text parser become the limit on several threads.