XWiki Platform
  1. XWiki Platform
  2. XWIKI-2874

Allow XWiki to scale with large number of objects

    Details

    • Type: Improvement Improvement
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7 M3
    • Fix Version/s: None
    • Component/s: Model
    • Labels:
    • keywords:
      patch
    • Pull Request Status:
      Awaiting Contributor feedback
    • Similar issues:
      XWIKI-750EditObjects should allow editing a specified object
      XWIKI-1415Use SVN revision number as XWiki's build number when displaying the version
      XWIKI-2938XWiki Objects' number field values are not preserved in previous versions of a page in case of object deletion
      XWIKI-7339Watchlist documents property does not scale
      XWIKI-6658The configured default image quality for scaled JPEGs is ignored
      XWIKI-2339Cannot import large attachments
      XWIKI-6530Scaled images are not exported to PDF
      XWIKI-334Impossible to create xwiki:properties with name equal to xwiki:object jcr properties
      XWIKI-7119Add a link to the class of an object on the object editor
      XWIKI-6072Reduce number of dependencies drawn transitively by the xwiki-velocity module

      Description

      It seems pages get very slow to load when documents have lots of objects attached to them. For example a page with 1000 comments will load very slowly.

      We need to find a strategy or practice (generic or not) that allows XWiki pages to load fast even when they have lots of objects.

      1. testNewDocumentLoader.groovy
        4 kB
        CalebJamesDeLisle
      2. XWIKI-2874-newLoadXWikiDoc-0.01.patch
        16 kB
        CalebJamesDeLisle
      3. XWIKI-2874-newLoadXWikiDoc-0.02.patch
        24 kB
        CalebJamesDeLisle

        Issue Links

          Activity

          Hide
          CalebJamesDeLisle added a comment -

          added javadoc.

          Fixed:

          • Error on loading document with 1 or more objects which have no properties (not including custom mapped props.)
          • Endless loop if the db gives us a property which doesn't fit any of our objects.

          If an empty ListProperty is gotten, when it is re saved hibernate checks the new version against the origional and for the origional it sends a BaseProperty to ListProperty.equals
          I think this is because the properties are gotten by a query for BaseProperty.
          It works with this added to ListProperty.equals

          if (list2 == null && obj.getClass().getName().indexOf(".BaseProperty")!=-1){
              list2 = new ArrayList<String>();
          }
          

          I would like a better solution, but this is all I can figure out at the moment, I think this issue is intrinsic to the fast method of loading by loading all properties in a single query.

          Show
          CalebJamesDeLisle added a comment - added javadoc. Fixed: Error on loading document with 1 or more objects which have no properties (not including custom mapped props.) Endless loop if the db gives us a property which doesn't fit any of our objects. If an empty ListProperty is gotten, when it is re saved hibernate checks the new version against the origional and for the origional it sends a BaseProperty to ListProperty.equals I think this is because the properties are gotten by a query for BaseProperty. It works with this added to ListProperty.equals if (list2 == null && obj.getClass().getName().indexOf(".BaseProperty")!=-1){ list2 = new ArrayList<String>(); } I would like a better solution, but this is all I can figure out at the moment, I think this issue is intrinsic to the fast method of loading by loading all properties in a single query.
          Hide
          CalebJamesDeLisle added a comment -

          I was able to solve the problem from inside of the new loader method, albeit with a workaround. The problem is that a prototype object is used as the id to load the real object, and when the object is saved, hibernate compares the id to see if the owner has changed (see: isOwnerUnchanged http://www.docjar.com/html/api/org/hibernate/event/def/ProxyVisitor.java.html ) Since the Id is a ListProperty, Hibernate must know that it is dealing with a ListProperty and not a BaseProperty. This only seems to matter when the list is empty so the workaround in 0.03 is as follows:

          if(thisProp.getClassType().indexOf(".DBStringListProperty")!=-1
             && ((List) thisProp.getValue()).size()==0){
              session.evict(thisProp);
              session.load(thisProp, thisProp);
          }
          

          It reloads the object, making sure hibernate knows that the object is a ListProperty and not a BaseProperty. It will hurt load speed, but I don't think it will be very severe.

          I also added a piece to xwiki.cfg which reads as follows:

          #-# New more scalable document loader. The new loader is faster at loading documents which have lots of objects,
          #-# however it is new and complex so it is turned off by default. To turn it on, uncomment the following line and set it to =1
          # xwiki.store.hibernate.useNewDocumentLoader=0
          

          This will activate and deactivate the new document loader.

          Show
          CalebJamesDeLisle added a comment - I was able to solve the problem from inside of the new loader method, albeit with a workaround. The problem is that a prototype object is used as the id to load the real object, and when the object is saved, hibernate compares the id to see if the owner has changed (see: isOwnerUnchanged http://www.docjar.com/html/api/org/hibernate/event/def/ProxyVisitor.java.html ) Since the Id is a ListProperty, Hibernate must know that it is dealing with a ListProperty and not a BaseProperty. This only seems to matter when the list is empty so the workaround in 0.03 is as follows: if(thisProp.getClassType().indexOf(".DBStringListProperty")!=-1 && ((List) thisProp.getValue()).size()==0){ session.evict(thisProp); session.load(thisProp, thisProp); } It reloads the object, making sure hibernate knows that the object is a ListProperty and not a BaseProperty. It will hurt load speed, but I don't think it will be very severe. I also added a piece to xwiki.cfg which reads as follows: #-# New more scalable document loader. The new loader is faster at loading documents which have lots of objects, #-# however it is new and complex so it is turned off by default. To turn it on, uncomment the following line and set it to =1 # xwiki.store.hibernate.useNewDocumentLoader=0 This will activate and deactivate the new document loader.
          Hide
          CalebJamesDeLisle added a comment -

          I have developed a test script to find problems with the new document loader. Just because the loader works on my installation of XE doesn't mean it will work with every database in existence. This test script depends on fixing the bug XWIKI-4377. loader0.03 (which is only a slight change from 0.02) does not pass this test so I will remove it.

          Show
          CalebJamesDeLisle added a comment - I have developed a test script to find problems with the new document loader. Just because the loader works on my installation of XE doesn't mean it will work with every database in existence. This test script depends on fixing the bug XWIKI-4377 . loader0.03 (which is only a slight change from 0.02) does not pass this test so I will remove it.
          Hide
          CalebJamesDeLisle added a comment -

          There is an email describing the change here:
          http://xwiki.markmail.org/message/bndsnhwh3saeoc5i

          Is it worthwhile to rework the loader to load multiple documents simultaneously? We could have a request batcher which holds requests and executes them periodically (say every 50ms.) It might improve performance on a large database.
          Is this worthwhile?

          Show
          CalebJamesDeLisle added a comment - There is an email describing the change here: http://xwiki.markmail.org/message/bndsnhwh3saeoc5i Is it worthwhile to rework the loader to load multiple documents simultaneously? We could have a request batcher which holds requests and executes them periodically (say every 50ms.) It might improve performance on a large database. Is this worthwhile?
          Hide
          Sergiu Dumitriu added a comment -

          XWIKI-8948 should allow to simplify the patch a bit.

          Show
          Sergiu Dumitriu added a comment - XWIKI-8948 should allow to simplify the patch a bit.

            People

            • Assignee:
              Unassigned
              Reporter:
              Vincent Massol
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Date of First Response: