Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 9.11.1
    • Fix Version/s: 9.11.5, 10.3
    • Component/s: Old Core
    • Labels:
      None
    • Environment:
      debian 9.3
    • Difficulty:
      Unknown
    • Documentation:
      N/A
    • Documentation in Release Notes:
      N/A
    • Similar issues:

      Description

      Running Jetty 9.4.6.v20170531 on a debian 9.3

      We recently moved from XWiki 6.4.4 to XWiki 9.9 then 9.11.1.

      6.4.4 to 9.9 migration hapened on Jan/15-16

      I think the issue started raising up starting from XWiki 9.11.1, which happened on Jan/31. We can observe that on the attached graph (sorry for the Y-axis ...)

      The number of opened file limit for the java process was historically 32768 (for all versions), and it has never reach it until two days ago where I raised to 65535.

      You can see on the graph that from a fresh start, it's doesn't always raise at the same rate and can reach the limit more or less rapidly.

      Now, I have inspected the number of opened files, here is a report with about 1 hours interval :

      time is 9:38
      
      number of *single* files (on filesystem) opened by the process : 906
      (sudo lsof -p 32287 | egrep '(REG|DIR)' | awk '{print $9'} | sort | uniq -c |sort -n | wc -l)
      
      number of handlers (all types) for the process : 13923
      (sudo lsof  -p 32287 | wc -l)
      
      partial handlers listing, by unique file
      (sudo lsof -p 32287 | egrep '(REG|DIR)' | awk '{print $9'} | sort | uniq -c |sort -n)
      
      ( ... < 23 )
           23 /appdata/webapps/root/resources/icons/xwiki/favicon144.png
           23 /appdata/webapps/root/resources/icons/xwiki/favicon16.png
           23 /appdata/webapps/root/resources/icons/xwiki/favicon.ico
           23 /appdata/webapps/root/resources/icons/xwiki/favicon.svg
           35 /appdata/webapps/root/resources/icons/datamodel/calendar.png
           36 /appdata/webapps/root/resources/uicomponents/widgets/datepicker/dateTimePicker.js
           39 /appdata/webapps/root/resources/icons/silk/arrow_refresh.png
           39 /appdata/webapps/root/resources/icons/silk/lock_edit.png
           39 /appdata/webapps/root/resources/icons/silk/page_white_copy.png
           39 /appdata/webapps/root/resources/icons/silk/page_white_edit.png
           40 /appdata/webapps/root/resources/icons/silk/bullet_arrow_up.png
           41 /appdata/webapps/root/resources/icons/silk/textfield_rename.png
           44 /appdata/webapps/root/resources/icons/silk/bullet_arrow_down.png
           63 /appdata/webapps/root/resources/icons/silk/user.png
           76 /appdata/webapps/root/resources/icons/xwiki/ajax-loader-large.gif
          183 /appdata/webapps/root/resources/icons/silk/link.png
          185 /appdata/webapps/root/resources/icons/silk/cancel.png
          185 /appdata/webapps/root/resources/icons/silk/delete.png
          185 /appdata/webapps/root/resources/icons/silk/eye.png
          185 /appdata/webapps/root/resources/icons/silk/textfield_add.png
          193 /appdata/webapps/root/resources/icons/silk/tag_yellow.png
          227 /appdata/webapps/root/resources/icons/xwiki/pagination-controls.png
          368 /appdata/webapps/root/resources/icons/silk/add.png
          368 /appdata/webapps/root/resources/icons/silk/comment.png
          368 /appdata/webapps/root/resources/icons/silk/comments_add.png
          368 /appdata/webapps/root/resources/icons/silk/folder.png
          368 /appdata/webapps/root/resources/icons/silk/note_add.png
          368 /appdata/webapps/root/resources/icons/silk/note.png
          368 /appdata/webapps/root/resources/icons/xwiki/accordion.png
          368 /appdata/webapps/root/resources/icons/xwiki/parent_edit.gif
          368 /appdata/webapps/root/resources/uicomponents/require/deferred.js
          369 /appdata/webapps/root/resources/icons/silk/attach.png
          369 /appdata/webapps/root/resources/icons/silk/bullet_add.png
          369 /appdata/webapps/root/resources/icons/silk/bullet_toggle_minus.png
          369 /appdata/webapps/root/resources/icons/silk/bullet_toggle_plus.png
          369 /appdata/webapps/root/resources/icons/silk/chart_organisation.png
          369 /appdata/webapps/root/resources/icons/silk/cross.png
          369 /appdata/webapps/root/resources/icons/silk/help.png
          369 /appdata/webapps/root/resources/icons/silk/page_white_text.png
          369 /appdata/webapps/root/resources/icons/silk/pencil.png
          369 /appdata/webapps/root/resources/icons/silk/tick.png
          369 /appdata/webapps/root/resources/icons/xwiki/external-link.png
          369 /appdata/webapps/root/resources/icons/xwiki/spinner.gif
          369 /appdata/webapps/root/resources/icons/xwiki/spinner-white.gif
          369 /appdata/webapps/root/resources/js/prototype/prototype.js
          369 /appdata/webapps/root/resources/js/xwiki/compatibility.js
          369 /appdata/webapps/root/resources/js/xwiki/eventsBridge.js
          369 /appdata/webapps/root/resources/js/xwiki/markerScript.js
          369 /appdata/webapps/root/resources/js/xwiki/meta.js
      
      time is 10:37
      number of *single* files (on filesystem) opened by the process : 914
      (sudo lsof -p 32287 | egrep '(REG|DIR)' | awk '{print $9'} | sort | uniq -c |sort -n | wc -l)
      
      number of handlers (all types) for the process : 13874
      (sudo lsof  -p 32287 | wc -l)
      
      partial handlers listing, by unique file
      (sudo lsof -p 32287 | egrep '(REG|DIR)' | awk '{print $9'} | sort | uniq -c |sort -n)
      
      ... (< 24)
           24 /appdata/webapps/root/resources/icons/xwiki/favicon144.png
           24 /appdata/webapps/root/resources/icons/xwiki/favicon16.png
           24 /appdata/webapps/root/resources/icons/xwiki/favicon.ico
           24 /appdata/webapps/root/resources/icons/xwiki/favicon.svg
           36 /appdata/webapps/root/resources/icons/datamodel/calendar.png
           37 /appdata/webapps/root/resources/uicomponents/widgets/datepicker/dateTimePicker.js
           39 /appdata/webapps/root/resources/icons/silk/arrow_refresh.png
           39 /appdata/webapps/root/resources/icons/silk/lock_edit.png
           39 /appdata/webapps/root/resources/icons/silk/page_white_copy.png
           39 /appdata/webapps/root/resources/icons/silk/page_white_edit.png
           40 /appdata/webapps/root/resources/icons/silk/bullet_arrow_up.png
           41 /appdata/webapps/root/resources/icons/silk/textfield_rename.png
           44 /appdata/webapps/root/resources/icons/silk/bullet_arrow_down.png
           66 /appdata/webapps/root/resources/icons/silk/user.png
           77 /appdata/webapps/root/resources/icons/xwiki/ajax-loader-large.gif
          186 /appdata/webapps/root/resources/icons/silk/link.png
          189 /appdata/webapps/root/resources/icons/silk/cancel.png
          189 /appdata/webapps/root/resources/icons/silk/delete.png
          189 /appdata/webapps/root/resources/icons/silk/eye.png
          189 /appdata/webapps/root/resources/icons/silk/textfield_add.png
          196 /appdata/webapps/root/resources/icons/silk/tag_yellow.png
          230 /appdata/webapps/root/resources/icons/xwiki/pagination-controls.png
          373 /appdata/webapps/root/resources/icons/silk/add.png
          373 /appdata/webapps/root/resources/icons/silk/comment.png
          373 /appdata/webapps/root/resources/icons/silk/comments_add.png
          373 /appdata/webapps/root/resources/icons/silk/note_add.png
          373 /appdata/webapps/root/resources/icons/silk/note.png
          373 /appdata/webapps/root/resources/icons/xwiki/accordion.png
          373 /appdata/webapps/root/resources/icons/xwiki/parent_edit.gif
          373 /appdata/webapps/root/resources/uicomponents/require/deferred.js
          374 /appdata/webapps/root/resources/icons/silk/attach.png
          374 /appdata/webapps/root/resources/icons/silk/bullet_add.png
          374 /appdata/webapps/root/resources/icons/silk/bullet_toggle_minus.png
          374 /appdata/webapps/root/resources/icons/silk/bullet_toggle_plus.png
          374 /appdata/webapps/root/resources/icons/silk/chart_organisation.png
          374 /appdata/webapps/root/resources/icons/silk/cross.png
          374 /appdata/webapps/root/resources/icons/silk/folder.png
          374 /appdata/webapps/root/resources/icons/silk/help.png
          374 /appdata/webapps/root/resources/icons/silk/page_white_text.png
          374 /appdata/webapps/root/resources/icons/silk/pencil.png
          374 /appdata/webapps/root/resources/icons/silk/tick.png
          374 /appdata/webapps/root/resources/icons/xwiki/external-link.png
          374 /appdata/webapps/root/resources/icons/xwiki/spinner.gif
          374 /appdata/webapps/root/resources/icons/xwiki/spinner-white.gif
          374 /appdata/webapps/root/resources/js/prototype/prototype.js
          374 /appdata/webapps/root/resources/js/xwiki/compatibility.js
          374 /appdata/webapps/root/resources/js/xwiki/eventsBridge.js
          374 /appdata/webapps/root/resources/js/xwiki/markerScript.js
          374 /appdata/webapps/root/resources/js/xwiki/meta.js
      

      It doesn't sound good to me that a single file gets that much handler opened on it, and that those numbers are keeping increasing.

      Regards,

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tmortagne Thomas Mortagne
                Reporter:
                mh Martin
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Date of First Response: