Details
-
Bug
-
Resolution: Fixed
-
Major
-
16.10.0
Description
Notification pre-filtering can get slow on wikis with ten thousands of users. This is noticeable when there are lots of events (not clear how many) and notifications aren't appearing anymore.
Checking thread dumps shows that the "User event dispatcher thread" frequently shows stack traces like the following in a thread dump:
"User event dispatcher thread" #291 java.lang.Thread.State: RUNNABLE at java.util.TreeMap.getEntry(TreeMap.java:357) at java.util.TreeMap.get(TreeMap.java:279) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:353) at org.apache.lucene.index.CodecReader.terms(CodecReader.java:132) at org.apache.lucene.index.FilterLeafReader.terms(FilterLeafReader.java:415) at org.apache.lucene.index.Terms.getTerms(Terms.java:41) at org.apache.lucene.index.TermStates.loadTermsEnum(TermStates.java:109) at org.apache.lucene.index.TermStates.get(TermStates.java:167) at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:186) at org.apache.lucene.search.TermQuery$TermWeight.scorerSupplier(TermQuery.java:122) at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:515) at org.apache.lucene.search.Weight.bulkScorer(Weight.java:165) at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:368) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:759) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:720) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:549) at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:275) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1871) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1707) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:710) at org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1696) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:422) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:467) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2908) at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:225) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:234) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:926) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:939) at org.xwiki.eventstream.store.solr.internal.SolrEventStore.search(SolrEventStore.java:843) at org.xwiki.eventstream.store.solr.internal.SolrEventStore.search(SolrEventStore.java:833) at org.xwiki.eventstream.internal.DefaultEventStore.search(DefaultEventStore.java:356) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.isPrefiltered(UserEventDispatcher.java:324) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.isMailPrefiltered(UserEventDispatcher.java:309) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatch(UserEventDispatcher.java:288) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatch(UserEventDispatcher.java:338) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatchInContext(UserEventDispatcher.java:259) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatch(UserEventDispatcher.java:216) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.prefilterEvent(UserEventDispatcher.java:191) at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.flush(UserEventDispatcher.java:175) at org.xwiki.notifications.notifiers.internal.UserEventDispatcherScheduler.run(UserEventDispatcherScheduler.java:124) at org.xwiki.notifications.notifiers.internal.UserEventDispatcherScheduler$$Lambda$2398/0x00007f9765827720.run() at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.lang.Thread.run(Thread.java:840)
An analysis of the code shows that these Solr searches are only there to avoid checking notification preferences when an event has already been pre-filtered and assigned to the user. In almost all cases, this query won't find any results and thus we'll check the user's notification preferences, anyway. It seems far more likely that most users aren't receiving a notification for the event and thus we should first check if the user is receiving a notification before checking if we have maybe already stored the user. That can only happen when XWiki was pre-filtering the event while it was restarted, so this should be super rare.