Description
The database schema updates and data migrations run when a cluster member opens its first transaction, and they are only synchronized inside that member's JVM. When several members start with migrations enabled (for example after an upgrade, or when they are started at the same time), each of them runs the Hibernate schema update at the same time and runs every data migration, so a data migration can be applied twice. The same happens with an empty database, which every member tries to initialize.
In addition, a member with migrations disabled (the usual setup for all but one member) refuses an outdated database and keeps refusing it until it's restarted, even once another member has migrated the database.
Finding from a Kubernetes test cluster (two threads of one member started the migrations within a few milliseconds), and from reading the code for the rest.
Proposal:
- the member which has something to do (empty database to initialize, or migrations enabled) takes a lock of the database server (PostgreSQL advisory lock, MySQL/MariaDB named lock) held on a dedicated connection, reads the database version again once it holds it, and skips what another member did meanwhile; the database server releases the lock if the member dies;
- the other members wait for the lock (as long as needed by default, with a log every minute, or at most store.migration.lockTimeout seconds);
- databases without such a lock keep the current behaviour, with a warning when remote observation is enabled;
- a member which does not migrate reads the version of an outdated database again (at most every 10 seconds) instead of refusing it until it restarts.
Attachments
Issue Links
- depends on
-
XWIKI-25299 Allow running the database migrations and the Distribution Wizard at startup and then stopping XWiki
-
- Open
-
- is related to
-
XWIKI-25298 Refuse to use a database migrated by a more recent version of XWiki
-
- Open
-
- relates to
-
XWIKI-7006 Accessing a store that is not migrated to the latest data version should not be allowed
-
- Closed
-
-
XWIKI-23435 Introduce a cluster aware lock
-
- Open
-