Uploaded image for project: 'XWiki Platform'
  1. XWiki Platform
  2. XWIKI-19225

Job logs and statuses can pile up and consume storage space forever

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 13.4.4
    • Fix Version/s: None
    • Component/s: Job
    • Labels:
      None
    • Difficulty:
      Unknown
    • Similar issues:

      Description

      Job stauses and logs are stored forever in the permanent data folder. Most job logs are small, but others can grow to considerable individual sizes, e.g.:

      /var/lib/xwiki/data/jobs/status/refactoring/rename# du -sh ./* | sort -h
      20K     ./1608075951786-139
      20K     ./1628761856593-430
      24K     ./1607978328270-834
      ...
      5.4M    ./1627854408906-216
      5.5M    ./1625778870166-170
      12M     ./1628713967069-376
      13M     ./1629196610774-606
      

      Also, some job types have a more transient nature than others and, despite that, can end up taking the most space:

      /var/lib/xwiki/data/jobs/status# du -sh ./* | sort -h
      4.0K    ./store.properties
      28K     ./EventStreamWikiCleanerJob
      44K     ./distribution
      88K     ./flavor
      108K    ./file-manager
      136K    ./initialization
      304K    ./uix
      704K    ./display
      1.3M    ./solr
      47M     ./wikicreation
      84M     ./extension
      212M    ./refactoring
      

      After a while, particularly for small storage restrictions, this can become a real problem.

      We need a way to clean up these useless leftovers, either after a certain period or after a certain size and we also need a way to perform the cleanup preferentially, as some job logs might be needed on the long run (like the distribution job status, signaling that DW has executed), while others need to be deleted (like rendering).

      Also, others do not necessarily grow (like wikicreation but can still end up consuming a considerable size without ever being actually useful and never being reclaimed. When the storage limit is 1GB, a 50MB wikicreation logs means 5%, which is a lot to drag around, including after upgrades.

      The current workaround is for admins to create either a cron task or scheduled job that should perform the cleanup periodically.

      A better solution would be if the product provides a configurable cleanup job with good defaults to address the most common issues.

      An even more advanced step would be to do XWIKI-19215 and give full control and overview to admins over the jobs history and its impact on used storage.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              enygma Eduard Moraru
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: