Details
-
Bug
-
Resolution: Solved By
-
Major
-
14.10.16
-
Unknown
-
N/A
-
N/A
-
Description
We observed complete lockups of XWiki instances due to a single write to the security cache ending in an infinite loop. This seems to happen mostly during Solr indexing, this can also be shortly after startup. Indicative is that when performing a thread dump, there is one thread that has stack trace similar to this one that contains BoundedLocalCache, it consumes 100% CPU on one core:
at java.util.concurrent.ConcurrentHashMap.get(java.base@11.0.21/ConcurrentHashMap.java:940) at com.github.benmanes.caffeine.cache.BoundedLocalCache.put(BoundedLocalCache.java:2025) at com.github.benmanes.caffeine.cache.BoundedLocalCache.put(BoundedLocalCache.java:1993) at org.infinispan.commons.util.AbstractDelegatingMap.put(AbstractDelegatingMap.java:51) at org.infinispan.container.impl.AbstractInternalDataContainer.put(AbstractInternalDataContainer.java:152) at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:157) at org.infinispan.statetransfer.CommitManager.commitEntry(CommitManager.java:137) at org.infinispan.statetransfer.CommitManager.commit(CommitManager.java:100) at org.infinispan.interceptors.locking.ClusteringDependentLogic$LocalLogic.commitSingleEntry(ClusteringDependentLogic.java:445) at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntryOrdered(ClusteringDependentLogic.java:336) at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:248) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:650) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:936) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:630) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:724) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyAndFixVersion(EntryWrappingInterceptor.java:789) at org.infinispan.interceptors.impl.EntryWrappingInterceptor$$Lambda$453/0x0000000840632040.apply(Unknown Source) at org.infinispan.interceptors.SyncInvocationStage.thenApply(SyncInvocationStage.java:44) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:784) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:338) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:63) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:155) at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lambda$nonTxLockAndInvokeNext$3(AbstractLockingInterceptor.java:318) at org.infinispan.interceptors.locking.AbstractLockingInterceptor$$Lambda$975/0x0000000840e6b040.apply(Unknown Source) at org.infinispan.interceptors.SyncInvocationStage.andHandle(SyncInvocationStage.java:69) at org.infinispan.interceptors.locking.AbstractLockingInterceptor.nonTxLockAndInvokeNext(AbstractLockingInterceptor.java:313) at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:138) at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:41) at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:82) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:63) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:155) at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:268) at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:224) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:63) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55) at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:61) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:63) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:128) at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:89) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:247) at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297) at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101) at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:83) at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1306) at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1834) at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:225) at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:449) at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:698) at org.xwiki.cache.infinispan.internal.InfinispanCache.set(InfinispanCache.java:94) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCache.addEntry(DefaultSecurityCache.java:707) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCache.add(DefaultSecurityCache.java:761) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCache.add(DefaultSecurityCache.java:731) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCache.add(DefaultSecurityCache.java:724) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.getRules(DefaultSecurityCacheLoader.java:464) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:201) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:157) at org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:243) at org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:169) at org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:162) at org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:119) at org.xwiki.security.authorization.internal.BridgeAuthorizationManager.hasAccess(BridgeAuthorizationManager.java:67) at org.xwiki.security.authorization.internal.DefaultContextualAuthorizationManager.hasAccess(DefaultContextualAuthorizationManager.java:122) at org.xwiki.security.authorization.internal.DefaultContextualAuthorizationManager.hasAccess(DefaultContextualAuthorizationManager.java:117) at org.xwiki.security.authorization.internal.XWikiCachingRightService.checkAccess(XWikiCachingRightService.java:273) at com.xpn.xwiki.XWiki.checkAccess(XWiki.java:4387) at com.xpn.xwiki.XWiki.prepareDocuments(XWiki.java:5825) at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:548) at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:339) at com.xpn.xwiki.web.LegacyActionServlet.service(LegacyActionServlet.java:114) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
Other threads are blocked at stack traces ending with
java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.21/Native Method) - parking to wait for <0x000000070272bac8> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.21/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.21/AbstractQueuedSynchronizer.java:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.21/AbstractQueuedSynchronizer.java:1009) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.21/AbstractQueuedSynchronizer.java:1324) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.21/ReentrantReadWriteLock.java:738) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCache.getEntry(DefaultSecurityCache.java:613) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCache.get(DefaultSecurityCache.java:864)
It is not clear if this has possibly been solved by the rewrite of the security cache to no longer require the storage of the hierarchy in the security cache (XWIKI-21039) which could trigger cascading deletes. The documentation of the RemovalListener says that "Implementations of this interface should avoid performing blocking calls or synchronizing on shared resources." - and prior to XWiki 15.6 we could empty the whole security cache in our implementation of this interface, which is clearly not what this documentation advises. The best case would be that this left the security cache in an invalid state and this infinite loop is the result of it as this would mean that this bug has been fixed by XWIKI-21039. Unfortunately, there are no clear reproduction steps such that we could verify that this is indeed the case. What seems to be common to the cases we observed so far is that content from Confluence has been imported, and the problem happened during indexing by Solr.
Increasing the size of the security cache seems to avoid this problem as it reduces evictions.
Attachments
Issue Links
- relates to
-
XWIKI-21039 Don't dispose security cache entries hierarchically when the cache is full
-
- Closed
-