Uploaded image for project: 'XWiki Platform'
  1. XWiki Platform
  2. XWIKI-10594

OOM errors thrown when uploading a lot of pages

    Details

    • Difficulty:
      Unknown
    • Similar issues:

      Description

      Hello,
      when uploading a lot of pages into XWiki 6.0.1 enterprise which is configured to run inside tomcat and use postgresql as a backend I'm still very easily able to nearly kill whole xwiki by uploading few thousands of small pages. It's java basically killed by various OOM errors thrown from random places.
      If you do have some time to duplicate, then create a content.xml with

      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
      <page xmlns="http://www.xwiki.org">
        <title>Semantic XWiki Benchmark page</title>
        <syntax>xwiki/2.0</syntax>
        <content>This is a page content.</content>
      </page>
      

      and then upload pages with:

      $ curl -X PUT -H "Content-Type: application/xml" -d @content.xml -o /dev/null http://Admin:admin@localhost:8080/xwiki/rest/wikis/xwiki/spaces/Benchmark/pages/BenchPage_[1-500000]
      

      it takes some time (hour(s)) to get into OOM so be patient. If this does not work for you, use higher number of pages or run in parallel with:

      #!/usr/bin/env bash
      echo "starting curl 1"
      curl -s -X PUT -H "Content-Type: application/xml" -d @content.xml -o /dev/null http://Admin:admin@localhost:8080/xwiki/rest/wikis/xwiki/spaces/Benchmark/pages/BenchPage_[1-125000] &
      echo "starting curl 2"
      curl -s -X PUT -H "Content-Type: application/xml" -d @content.xml -o /dev/null http://Admin:admin@localhost:8080/xwiki/rest/wikis/xwiki/spaces/Benchmark/pages/BenchPage_[125001-250000] &
      echo "starting curl 3"
      curl -s -X PUT -H "Content-Type: application/xml" -d @content.xml -o /dev/null http://Admin:admin@localhost:8080/xwiki/rest/wikis/xwiki/spaces/Benchmark/pages/BenchPage_[250001-375000] &
      echo "starting curl 4"
      curl -s -X PUT -H "Content-Type: application/xml" -d @content.xml -o /dev/null http://Admin:admin@localhost:8080/xwiki/rest/wikis/xwiki/spaces/Benchmark/pages/BenchPage_[375000-500000] &
      echo "waiting for jobs to complete..."
      wait
      

      Of course, you will probably tell me that I shall increase java's heap limit, then believe me, I've duplicated this with:

      CATALINA_OPTS="-d64 -server -Xms2048m -Xmx2048m -XX:MaxPermSize=196m -Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC -XX:MaxGCPauseMillis=100"
      export CATALINA_OPTS
      

      in setenv.sh in catalina's bin directory and also with 8GB RAM setup confidure with:

      CATALINA_OPTS="-d64 -server -Xms8192m -Xmx8192m -XX:MaxPermSize=196m -Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC -XX:MaxGCPauseMillis=100"
      export CATALINA_OPTS
      

      everything is running on 32GB RAM box with Solaris 11.1 OS and java 1.7.0_07-b10

      The trace from one run of the issue looks:

      Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
              at java.io.BufferedInputStream.<init>(BufferedInputStream.java:195)
              at java.io.BufferedInputStream.<init>(BufferedInputStream.java:175)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:697)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
              at java.lang.Thread.run(Thread.java:722)
      Exception in thread "ajp-bio-8009-AsyncTimeout" java.lang.OutOfMemoryError: Java heap space
      Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space
              at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1427)
              at java.util.AbstractCollection.toArray(AbstractCollection.java:179)
              at org.apache.catalina.session.StandardSession.keys(StandardSession.java:1765)
              at org.apache.catalina.session.StandardSession.expire(StandardSession.java:863)
              at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:656)
              at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:532)
              at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:517)
              at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1352)
              at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1530)
              at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
              at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
              at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1519)
              at java.lang.Thread.run(Thread.java:722)
      Exception in thread "Lucene Index Updater" java.lang.OutOfMemoryError: Java heap space
      Jun 26, 2014 7:10:26 PM ServerCommunicatorAdmin reqIncoming
      WARNING: The server has decided to close this client connection.
      2014-06-26 19:10:31,992 [XWiki Solr index thread] ERROR o.a.s.c.SolrCore               - java.lang.NullPointerException
              at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3119)
              at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2892)
              at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1210)
              at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1690)
              at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
              at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:518)
              at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:618)
              at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2891)
              at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3049)
              at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3016)
              at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:578)
              at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
              at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
              at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1458)
              at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1435)
              at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
              at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916)
              at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:150)
              at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:118)
              at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
              at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:146)
              at org.xwiki.search.solr.internal.AbstractSolrInstance.commit(AbstractSolrInstance.java:101)
              at org.xwiki.search.solr.internal.DefaultSolrIndexer.commit(DefaultSolrIndexer.java:441)
              at org.xwiki.search.solr.internal.DefaultSolrIndexer.processBatch(DefaultSolrIndexer.java:429)
              at org.xwiki.search.solr.internal.DefaultSolrIndexer.runInternal(DefaultSolrIndexer.java:374)
              at com.xpn.xwiki.util.AbstractXWikiRunnable.run(AbstractXWikiRunnable.java:131)
              at java.lang.Thread.run(Thread.java:722)
       
      Exception in thread "XWiki Solr index thread" java.lang.NullPointerException
              at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3119)
              at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2892)
              at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1210)
              at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1690)
              at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
              at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:121)
              at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)
              at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
              at ch.qos.logback.classic.Logger.error(Logger.java:559)
              at org.xwiki.search.solr.internal.DefaultSolrIndexer.commit(DefaultSolrIndexer.java:443)
              at org.xwiki.search.solr.internal.DefaultSolrIndexer.processBatch(DefaultSolrIndexer.java:429)
              at org.xwiki.search.solr.internal.DefaultSolrIndexer.runInternal(DefaultSolrIndexer.java:374)
              at com.xpn.xwiki.util.AbstractXWikiRunnable.run(AbstractXWikiRunnable.java:131)
              at java.lang.Thread.run(Thread.java:722)
      Exception in thread "Lucene Merge Thread #10103" org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
              at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
              at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      

      but as I said, it's quite random. On xwiki-devs@ mailing list I've already reported several others (notifier was also considered to be a culprit here).
      Thanks!
      Karel

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kgardas Karel Gardas
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Date of First Response: