Uploaded image for project: 'XWiki Platform'
  1. XWiki Platform
  2. XWIKI-17825

Listener DatabaseMailResender is trying to resend the stuck emails at each restart creating an OOM and loop that prevents XWiki to properly restart

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Solved By
    • Affects Version/s: 8.4.6
    • Fix Version/s: None
    • Component/s: Mail
    • Labels:
      None
    • Environment:
      8.4.6 + backport for XWIKI-13991
    • Difficulty:
      Unknown
    • Documentation:
      N/A
    • Documentation in Release Notes:
      N/A
    • Similar issues:

      Description

      We noticed that following a backport for XWIKI-13991 on a 8.4.6 XWiki the server sometimes gets stuck in a loop during a restart. The wiki sends a lot of emails using a scheduler job.
      The error from the logs during the restart loop

      2020-09-08 17:13:23,326 [XWiki initialization] WARN o.x.m.i.DatabaseMailResender - Failed to load mail content for batchId [5a3813da-1632-433b-90e4-c2f946ef597f], messageId [WVy8QUmH3iFNIQa0WLkRTAgBxU0=]

      Other details from the server logs:

      • Java heap space
        SEVERE: 
        java.lang.OutOfMemoryError: Java heap space
        
        Sep 08, 2020 2:03:49 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor doRun
        SEVERE: 
        java.lang.OutOfMemoryError: Java heap space
        	at java.io.BufferedWriter.<init>(BufferedWriter.java:105)
        	at java.io.BufferedWriter.<init>(BufferedWriter.java:88)
        	at java.io.PrintWriter.<init>(PrintWriter.java:148)
        	at javax.script.SimpleScriptContext.<init>(SimpleScriptContext.java:93)
        	at org.xwiki.script.internal.ScriptExecutionContextInitializer.initialize(ScriptExecutionContextInitializer.java:52)
        	at org.xwiki.context.internal.DefaultExecutionContextManager.runInitializers(DefaultExecutionContextManager.java:115)
        	at org.xwiki.context.internal.DefaultExecutionContextManager.initialize(DefaultExecutionContextManager.java:103)
        	at org.xwiki.container.servlet.internal.DefaultServletContainerInitializer.initializeRequest(DefaultServletContainerInitializer.java:114)
        	at com.xpn.xwiki.web.XWikiAction.initializeContainerComponent(XWikiAction.java:655)
        	at com.xpn.xwiki.web.XWikiAction.initializeXWikiContext(XWikiAction.java:640)
        	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:206)
        	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
        	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
        	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
        	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        	at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
        	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        	at org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
        	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
        	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
        	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        
        Sep 08, 2020 2:04:30 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
        SEVERE: 
        java.lang.OutOfMemoryError: Java heap space
        
        Sep 08, 2020 2:04:22 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
        SEVERE: 
        java.lang.OutOfMemoryError: Java heap space
        
        Sep 08, 2020 2:07:39 PM org.apache.tomcat.util.net.NioEndpoint$Acceptor run
        SEVERE: 
        java.lang.OutOfMemoryError: Java heap space
        
      • GC overhead limit exceeded
        Sep 08, 2020 4:16:14 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
        SEVERE: 
        java.lang.OutOfMemoryError: GC overhead limit exceeded
        	at java.util.Arrays.copyOf(Arrays.java:3332)
        	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        	at java.lang.StringBuilder.append(StringBuilder.java:136)
        	at org.apache.tomcat.util.threads.TaskThreadFactory.newThread(TaskThreadFactory.java:42)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
        	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
        	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
        	at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:166)
        	at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:146)
        	at org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:759)
        	at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1299)
        	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1252)
        	at java.lang.Thread.run(Thread.java:748)
        
        Sep 08, 2020 4:16:21 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor doRun
        SEVERE: 
        java.lang.OutOfMemoryError: GC overhead limit exceeded
        
        Sep 08, 2020 4:16:24 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
        SEVERE: 
        java.lang.OutOfMemoryError: GC overhead limit exceeded
        
        Sep 08, 2020 4:16:30 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor doRun
        SEVERE: 
        java.lang.OutOfMemoryError: GC overhead limit exceeded
        
        
      • then
        Sep 08, 2020 4:25:16 PM org.apache.catalina.core.StandardServer await
        INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
        Sep 08, 2020 4:25:22 PM org.apache.catalina.startup.VersionLoggerListener log
        INFO: Server version:        Apache Tomcat/7.0.79
        
        Sep 08, 2020 4:25:22 PM org.apache.catalina.startup.VersionLoggerListener log
        INFO: Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
        Sep 08, 2020 4:25:22 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
        INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
        
        

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              vmassol Vincent Massol
              Reporter:
              oana.tabaranu Elena-Oana Florea
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response: