Description
The job log UI currently renders the whole log at once. job_macros.vm#printStatusLog iterates the entire LogTail, and logging_macros.vm#printLog writes every entry into the page - including every frame of every stack trace, folded by CSS but present in the HTML. There is no pagination, no filtering and no size limit anywhere. That is what makes XWIKI-16647 (OOM when displaying a job with a huge log) possible, and what XWIKI-22653 reports.
Proposal: display the job log through a Live Data, with a dedicated Live Data source backed by the job status log tail.
Why it fits well
The file-backed log tail is already a random-access, indexed store. AbstractFileLoggerTail keeps an in-memory index of (byte position, log level) per event, loaded from a companion .index file, so:
- size() is O(1) and gives the Live Data entry count
- getLogEvent(int) is a seek() plus one deserialization, and maps directly onto the Live Data entry id
- filtering by level needs the index only, with no deserialization at all
- sorting by timestamp is free, since the tail is chronological
The result is that displaying a page of the log costs O(page) instead of O(whole log), which is the part that addresses XWIKI-16647.
Known gaps to handle
- LogTail#getLogEvents(LogLevel, offset, limit) cannot express "page N of the events at level X or above". In AbstractFileLoggerTail the offset and the limit both apply to the unfiltered index range, so a page of 15 can come back with 2 entries; in LogQueue the offset is unfiltered but the limit applies to the filtered result. There is also no count-by-level, only hasLogLevel(). Since the file index already carries the level, adding this to the commons API is cheap, but it is an API addition and needs its own XCOMMONS issue.
- Filtering or sorting on the message column requires deserializing every event, which is exactly the full scan to avoid. Either declare that column non-filterable and non-sortable, or wait for a log storage that can do it (XCOMMONS-3249).
- Rendering a log message as an HTML cell means reusing or replacing logging_macros.vm#printLogMessage, which does per-argument type dispatch (entity reference to document link, extension id to extension link, and so on) and carries its own TODO asking for a proper Java framework.
- Live Data is a paginated table, not an append stream, whereas the current UI polls and appends new entries. A running job would either poll updateEntries() on the current page, or keep the existing display until the job ends.
- Authorization must live in the Live Data entry store rather than in the calling template, because a Live Data source is reachable through the generic Live Data REST endpoint with arbitrary source parameters.
Related work
The server side already has an issue: XWIKI-22557 asks for offset and limit parameters on the /rest/joblog/<jobId> REST resource, which today supports only level and fromLevel.
This came out of the forum discussion at https://forum.xwiki.org/t/logging-warning-stacktraces-in-debug-mode/18728 (posts 9 to 11), where a paginated Live Data display was preferred over throttling repeated stack traces at capture time. Note that the full benefit depends on a job log storage that can paginate and filter efficiently, which is XCOMMONS-3249.
Attachments
Issue Links
- blocks
-
XCOMMONS-3738 Pass the exception to warn() and decide the stack trace display at rendering time
-
- Open
-
- relates to
-
XWIKI-22653 Don't render log items if they are not displayed
-
- Open
-
-
XWIKI-16647 Possible OOM when displaying a job with a huge log
-
- Reopened
-
-
XCOMMONS-3249 Allow storing job status and logs in a shared location
-
- Open
-
-
XWIKI-22557 Add an offset and a limit parameters to job log REST API
-
- Open
-
-
XWIKI-24665 Apply the logging best practices across commons, rendering and platform
-
- Closed
-