Live emails notifications should be grouped by XWiki pages

About the number of emails, we had this conversation on #xwiki:

gdelhumeau:
Quick question about live notifications emails
When we enable the live notifications emails, there is 2 possibles behaviours
1 - An email is sent for each document that has been modified. These emails can be grouped together in the client of the user (example: mail thread in gmail)
2 - A unique email is sent with all notifications that have happened during the last 10 minutes (if there are such events), and maybe an other one will be sent 10 minutes later with new notifications
Case 1: gmail can grouped emails that concern the same document in a unique mail thread
Case 2: we make sure we don't have more than 1 email per 10 minutes (the grace period), but grouping is more complicated since it can concern several documents
Currently we have 1), except that we miss the MessageID field to make gmail group things efficiently

tmortagne:
1 - ou mean for each DocumentUpdatedEvent ?
s/ou/you/

gdelhumeau
But I thought we had 2
Thomas: no, for each event. The grace period will make sure you don't receive more than 1 mail per 10 minutes concerning this document
But you can receive several emails during these 10 minutes if they concern different docs

tmortagne
talking about 1, not 2

gdelhumeau
in 1, we already have a grace period
I did not mention it for trying to make things easier

tmortagne
so you still have to wait for 10 minutes in both cases ?

gdelhumeau
Yes
Actually let me rephrase
1 - One mail per document, but not more than 1 mail per 10 minutes concerning this document
2 - One mail per 10 minutes, possibly concerning more than 1 document
Apparently the Watchlist did 1)

tmortagne
if we are waiting anyway I would be for grouping everything that can be grouped (but yes it's probably more work). I guess we could also let the user choose with profile preference

gdelhumeau
Clément Aubin: please confirm I'm not telling something wrong
*saying

tmortagne
so +1 for 2 but I prefer both 1 and 2 (with 2 being the default)

gdelhumeau
haha, it seems both
it *means

tmortagne
you have 1 already anyway so if you implement 2 it would be a pitty to ditch it

gdelhumeau
but you prefer 2 if I cannot do both, of course

tmortagne
yes
but it's not like I was -1 for 1 either
I'm fine with both, just prefer 2

gdelhumeau
vmassol, Enygma, Marius Dumitru Florea, evalica: please take 2 minutes to look at it

tmortagne
is the grace period configurable already ?

gdelhumeau
it is

tmortagne
ok
can I put something lower than 10 minutes
?

gdelhumeau
minimum is 0

tmortagne
what does 0 will do in practice ? constently execute sql queries to see if something new came up ?

enygma
Guillaume Delhumeau: you had a look at the realtime watchlist notification, right?

gdelhumeau
Enygma: I looked at your commits about Message-Id and so on (https://github.com/xwiki/xwiki-platform/commit/e38a0db8b93a1b00ced5f35a8e3ab45b5aa2b253)
I mean https://github.com/xwiki/xwiki-platform/commit/fd93f5cf2cd45444183bdd95f905d463aabae6f1

enygma
I personally prefer 1, since that was the point of real-time notifications, AFAIU. Also, coupling this with the email client grouping results in a really nice an useful overview of the received emails, allowing to focus on the changed pages. This way, in the email client, if you are watching 5 documents, you will never have more than 5 conversations to read. If you send batches, the best this we could do is to group all those batches under the same conversation ID, so that in the email client you only have 1 conversation, but it might get hard to follow at some point. Also, Gmail automatically breaks a conversation longer than X (around 100, AFAIR) messages.

tmortagne
I would prefer 1 too if you did not had to wait for 10 minutes

enygma
sure, if 1 also has the grace period / time buffer of around 5 minutes (maybe 10), that would be OK to. This is what we have now, AFAIU.
yes, 10 mins sounds a bit too much

tmortagne
but since in both cases we have to wait then lets group everything that can be grouped

enygma
I would even argue that it should be around 30 sec to 1 minute
and it should be a moving window, based on incoming events
i.e. high traffic -> longer wait time to get notified of a bigger batch
small traffic -> faster notification of a smaller batch

tmortagne
personnaly I'm find iwith 2 and I will lower a lot the grace period n my instances
so be closer to real time

enygma
what I don't want to see is a notification mail about more than 1 document
(my preference)
because then the email client grouping can not be done
but heck, we could end up with everything configurable (and ages to implement )

tmortagne
yeah maybe, did not tough much about that, in my mind it was about getting a mail and delete it
I don't think about it like a conversation I need to group like forum notifications

gdelhumeau
ok so at least we need to set the ConversationId (that we don't have now)

tmortagne
but I can understand that after a while it's nicer
anyway if we keep 1 then I'm cool with that but if we do 2 seems to be configuration does not really add much work (but if it's not doable so be it)

vmassol
configurable 1 and 2 is fine with me too
if we have to choose one, I don't have much of an opinion ATM
both are fine with me

tmortagne
in any case we have to choose a default
but if everyone is find with 1 looks like a lot less work
*fine

vmassol
realtime is more 1 probably

tmortagne
we can always do 2 later

vmassol
(also 1 cost more in term of perf)
(more emails)

tmortagne
not sure about that
more email but less work to group things
so more work for the mail server, not us

vmassol
I was talking scalability of emails
I'm not sure how well or rather how bad realtime notif is when you have lots of users
imagine 100K users
and some activity
I'm not even sure there's enough time to send all the emails

tmortagne
you usually don't have 100k users listening to everything
so you almost never send that many emails

Enygma
well, during the night they will eventually get the emails

tmortagne
and the difference between 1 and 2 is probably something like sendig 1 email instead of a few most of the time
(per user listening to those changes)

vmassol
So one question could be: how many users using realtime do we need to make the sending not "realtime", i.e. taking longer than, say, 10 minutes
(knowing that we have a wait perdio before sending each mail, by default it's 8 seconds I think)

tmortagne
what wait period are you talking about ?

vmassol
yes it's 8s (just checked)

tmortagne
the mail module wait for 8s between each mail ?

vmassol
yes
"The delay to wait between each mail being sent, in milliseconds (defaults to 8 seconds)"

tmortagne
why ?

vmassol
to not be considered a spammer
the value wasn't chosen randomly
I read the doc on several mail server to find the value that would not consider XWiki a spammer
it's configurable ofc

tmortagne
so this is only when there is several mails, right ?
if you send a mail it's sent right away ?

vmassol
yes

tmortagne
so it's not "before sending each mail"

vmassol
(when thre are several to be sent in the queue)
So one question could be: how many users using realtime do we need to make the sending not "realtime", i.e. taking longer than, say, 10 minutes

(knowing that we have a wait perdio before sending each mail, by default it's 8 seconds I think)
^^^so there'll be several in the queue

tmortagne
the main criteria is not really teh number of users I think, it all depends what those user are watchling, no ?
you are not going to send all changes to all users all the time, right ?
S
vmassol
it should depend on users, activity in the wiki and indeed what users are watching

tmortagne
in any case the difference between 1 and 2 in number of email is probably not going to be very big most of the time unless the edit activity is very high and wide

vmassol
yes, my question was not so much for 1 and 2 but more in general

tmortagne
not sure what is you question then ? should we do real time notifications ?
IMO as long as it does not take much resources mail will arrive when they arrive

vmassol
so 10*60/8=75, so we can serve 80 emails per 10 minutes.
so if we have 80 users all folllowing one doc, the last reatime email will arrive after 10 minutes
(at best)
(then ofc if you're using this internally with your own mail server you can reduce the wait time to 0s)
(and then it's about how many emails we can prepare and send per seconds)
(I don't know this figure)
anyway, back to clover work, I've found a way to compare the reports I think

tmortagne
Guillaume Delhumeau: so looks like 1 with proper conversation ids