Pending live notification mails are sent again by each cluster member that starts

Finding from the Kubernetes test cluster, from code reading: the duplicates were not reproduced.

What happens

Each live notification mail waits in the memory of the cluster member that pre-filtered its event until the grace time (notifications.emails.live.graceTime, 10 minutes by default) has passed since the event date. The mail entity stored with the event is deleted only when the mail is prepared, so that a member which stops before sending can send it again when it starts: on ApplicationReadyEvent (main wiki) and on each WikiReadyEvent, PrefilteringLiveNotificationEmailListener starts MissingLiveNotificationMailsJob, which queues every event that still has a mail entity for a user with the LIVE interval.

That job has no filter on the cluster member: it also queues the events pre-filtered by the other members, whose mails are still waiting in their memory. So in a cluster, a member that starts within the grace time of an event queued on another member sends the same mail again, and a rolling restart of N members within the grace time of an event can send up to N+1 copies of the mail to each user. There is no configuration to avoid it, since the resume cannot be disabled.

UserEventDispatcher has the same problem on a smaller scale: every member pre-filters (and creates the mail entities of) the events that have no remote observation identifier, while the events that have one are pre-filtered only by the member that produced them.

Proposed change

• Each member resumes at startup only the pending mails of the events produced by itself (event remote observation identifier equal to observation.remote.id), that is the mails its previous run did not send. This needs a stable observation.remote.id, which is already the case when the permanent directory is kept or the identifier is configured.
• The cluster singleton node (XWIKI-25253) also resumes, for all the wikis, the pending mails of the events without remote observation identifier, at startup and when it becomes the singleton node.
• UserEventDispatcher pre-filters the events without remote observation identifier only on the cluster singleton node, and pre-filters the waiting events when the instance becomes the singleton node.
• When remote observation is disabled, nothing changes: all the pending mails are resumed and all the events are pre-filtered, as before.

The pending mails of a member that never comes back with the same identifier are no longer sent by the other members (they used to be sent, several times); picking them up after a delay can be handled separately.