A JAR extension upgrade makes its components unavailable until the end of the job, breaking concurrent requests with cryptic errors

Problem

When a JAR extension is upgraded, the components of the previous version are unregistered immediately by JarExtensionHandler.upgrade(), while JarExtensionHandler.install() deliberately does not register the components of the new version:

// If there is any JAR upgrade in progress, there is a high chance that the new JAR does not work well until the
// end of the job when the classloader will be reloaded. So we just skip loading the new JAR and its components,
// they will be loaded at the end of the job.
if (!this.jarExtenssionContext.isReloadRequired(namespace)) {
    initializeInternal(localExtension, namespace);
}

The new components only show up at the very end of the job, when JarExtensionJobFinishingListener drops the classloader(s) and re-initializes every JAR extension of the namespace on JobFinishingEvent.

This means that between the moment the first JAR of the extension is unregistered and the end of the whole install job (which can last a long time on a big instance, see XWIKI-24332), the extension's components simply do not exist, while XWiki keeps serving HTTP requests as usual. Any request reaching such a component during that window fails, and it fails with an error that tells neither the user nor the administrator reading the log what the actual cause is.

Concrete example

Upgrading the OpenID Connect Authenticator (2.25.4 to 2.26.0) from the Extension Updater on a 17.10.1 instance. As soon as the oidc-authenticator-configuration JAR had been upgraded, incoming requests started failing with a HTTP 500 and the log below. The Extension Updater UI displayed Failed to retrieve extension data. Server Error, its own progress-polling request being one of the failing ones.

WARN  c.x.x.w.XWikiAction  - Uncaught exception: Error number 0 in 11: Uncaught exception
com.xpn.xwiki.XWikiException: Error number 0 in 11: Uncaught exception
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:662)
	...
Caused by: org.infinispan.commons.IllegalLifecycleStateException: ISPN000323: Cache 'oidc.client.configuration' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
	at org.xwiki.cache.infinispan.internal.InfinispanCache.get(InfinispanCache.java:97)
	at org.xwiki.contrib.oidc.auth.internal.store.OIDCClientConfigurationCache.get(OIDCClientConfigurationCache.java:97)
	at org.xwiki.contrib.oidc.auth.internal.store.DefaultOIDCClientConfigurationStore.getOIDCClientConfiguration(DefaultOIDCClientConfigurationStore.java:128)
	at org.xwiki.contrib.oidc.auth.internal.OIDCClientConfiguration.getProperty(OIDCClientConfiguration.java:687)
	at org.xwiki.contrib.oidc.auth.OIDCAuthServiceImpl.getAuthenticator(OIDCAuthServiceImpl.java:98)
	at com.xpn.xwiki.user.impl.xwiki.XWikiAuthServiceImpl.checkAuth(XWikiAuthServiceImpl.java:198)
	at com.xpn.xwiki.XWiki.checkAuth(XWiki.java:4374)
	at org.xwiki.security.authorization.internal.XWikiCachingRightService.checkAccess(XWikiCachingRightService.java:268)
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:548)

What happened: unregistering the components of the previous version disposed OIDCClientConfigurationCache, whose dispose() removes the oidc.client.configuration cache from the Infinispan cache manager. The authenticator instance still referenced by XWiki#authService then kept using that dead cache.

The authenticator is the worst possible case, because XWiki#checkAuth runs for every single request (guests included), so the whole wiki answers 500 for the remainder of the job. XWIKI-16909 already covers the post-job part of this problem: XWiki#onJobFinished re-instantiates the configured authenticator on JobFinishedEvent, which is why the instance recovers by itself once the job ends and why no restart is needed. What is missing is the behaviour during the job.

Nothing here is specific to OIDC or to authenticators: the same window breaks any component provided by a JAR extension while that extension is being upgraded. See also XCOMMONS-751, XWIKI-18965 and XWIKI-18877 for other consequences of the same reload window.

Expected

Two directions have been discussed:

• Better error reporting. When a component cannot be found, or is dead, because the JAR providing it is in the middle of a reload, the error should say exactly that, instead of surfacing an unrelated-looking IllegalLifecycleStateException, NoClassDefFoundError or Uncaught exception. The information is available: the JAR handler already consults isReloadRequired(namespace) to decide whether to postpone the loading of the new JAR.
• Blocking incoming requests for the duration of the extension job, so that requests are explicitly held or rejected with a meaningful message rather than served against a half-unregistered component graph. This has been discussed several times already.

Note that silently falling back to another implementation is not wanted, at least not for authenticators: failing is better than authenticating users with an unexpected authenticator.