Uploaded image for project: 'XWiki Platform'
  1. XWiki Platform
  2. XWIKI-25297

Prevent several cluster members from migrating the database at the same time

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • None
    • 18.8.0
    • Old Core, Storage
    • Unknown

    Description

      The database schema updates and data migrations run when a cluster member opens its first transaction, and they are only synchronized inside that member's JVM. When several members start with migrations enabled (for example after an upgrade, or when they are started at the same time), each of them runs the Hibernate schema update at the same time and runs every data migration, so a data migration can be applied twice. The same happens with an empty database, which every member tries to initialize.
      In addition, a member with migrations disabled (the usual setup for all but one member) refuses an outdated database and keeps refusing it until it's restarted, even once another member has migrated the database.
      Finding from a Kubernetes test cluster (two threads of one member started the migrations within a few milliseconds), and from reading the code for the rest.
      Proposal:

      • the member which has something to do (empty database to initialize, or migrations enabled) takes a lock of the database server (PostgreSQL advisory lock, MySQL/MariaDB named lock) held on a dedicated connection, reads the database version again once it holds it, and skips what another member did meanwhile; the database server releases the lock if the member dies;
      • the other members wait for the lock (as long as needed by default, with a log every minute, or at most store.migration.lockTimeout seconds);
      • databases without such a lock keep the current behaviour, with a warning when remote observation is enabled;
      • a member which does not migrate reads the version of an outdated database again (at most every 10 seconds) instead of refusing it until it restarts.

      Attachments

        Issue Links

          Activity

            People

              ciaens Clément Christiaens
              ciaens Clément Christiaens
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: