The instances of a cluster don't share the filesystem blob store in the docker test framework

Problem

When a functional test runs several XWiki instances (@XWikiInstances(2)) with the filesystem blob store explicitly requested (-Dxwiki.test.ui.blobStore=filesystem), each instance gets its own filesystem blob store: the instances share the database but not the data stored in the blob store (attachments, deleted attachments, content of deleted documents). This is not a valid cluster setup, and it makes the cluster tests fail to read on one instance what was stored by the other.

The test framework already uses the S3 blob store for clusters by default (TestConfiguration#getBlobStore(): DEFAULT resolves to S3 when isCluster()), but an explicit filesystem applies to cluster tests too. The XWiki Environment Tests job passes it for its "Filesystem" environments (currently MySQL and Oracle).

Example

DocumentCacheIT (xwiki-platform-observation-test-docker) logs, in the MySQL and Oracle environments of XWiki Environment Tests (master builds 1406 and 1409), never in the S3 environments (MariaDB, PostgreSQL) nor in the main job:

ERROR i.o.r.c.DocumentEventConverter - Failed to restore deleted document [xwiki:Test.CacheSync]
com.xpn.xwiki.XWikiException: Error number 2002 in 2: Cannot find any content for the deleted document [Test.CacheSync ()]
	at com.xpn.xwiki.doc.XWikiDeletedDocument.restoreDocument(XWikiDeletedDocument.java:356)
	at com.xpn.xwiki.store.XWikiHibernateRecycleBinStore.restoreFromRecycleBin(XWikiHibernateRecycleBinStore.java:314)
	at com.xpn.xwiki.internal.observation.remote.converter.DocumentEventConverter.unserializeDeletedDocument(DocumentEventConverter.java:133)
	at com.xpn.xwiki.internal.observation.remote.converter.DocumentEventConverter.fromRemote(DocumentEventConverter.java:100)
	...
ERROR c.x.x.d.XWikiDocument - Error while trying to load deleted attachment [parentReference = [xwiki:Test.AttachementCacheSync], filename = [file.ext], version = [1.1]] for doc [Test.AttachementCacheSync]
com.xpn.xwiki.XWikiException: Error number 3002 in 3: The attachment [Attachment xwiki:Test.AttachementCacheSync@file.ext] (file ...) could not be found in the filesystem attachment store.
ERROR .DocumentSolrMetadataExtractor - Failed to retrieve the content of attachment [Attachment xwiki:Test.AttachementCacheSync@file.ext]

The document is deleted on the first instance: the recycle bin entry is in the shared database, but the content of the deleted document is in the blob store of the first instance. When the second instance receives the remote deletion event, DocumentEventConverter restores the deleted document from the recycle bin to provide it in the local event, and cannot find its content. Same for the deleted attachment and the Solr indexing of the attachment.

Proposed solution

When a test runs a cluster with the filesystem blob store, make all the instances use the same blob store directory. The platform uses a single filesystem blob store, store/file, located in <permanent directory>/store/file (not configurable):

• containerized servlet engines: bind-mount a shared directory of the host (e.g. <test output directory>/blobstore) on <permanent directory>/store/file in the container of each instance (ServletContainerExecutor, next to the optional mount of the permanent directory; the containers run as root)
• Jetty standalone (each instance runs from its own directory on the host): make <permanent directory>/store/file of each instance a symbolic link to the shared directory before starting it

Alternative: resolve an explicit filesystem blob store to S3 for clusters, but then the "Filesystem" environments would not test the filesystem blob store in a cluster anymore.

The cluster tests are DocumentCacheIT, InstalledExtensionIndexIT, PanelsIT and RemoteObservationIT.