Uploaded image for project: 'XWiki Platform'
  1. XWiki Platform
  2. XWIKI-14844

Live emails notifications should be grouped by XWiki pages

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 9.10-rc-1
    • Fix Version/s: None
    • Component/s: Notifications
    • Labels:
      None
    • Difficulty:
      Unknown
    • Similar issues:

      Description

      About the number of emails, we had this conversation on #xwiki:

      Guillaume Delhumeau:
      Quick question about live notifications emails
      When we enable the live notifications emails, there is 2 possibles behaviours
      1 - An email is sent for each document that has been modified. These emails can be grouped together in the client of the user (example: mail thread in gmail)
      2 - A unique email is sent with all notifications that have happened during the last 10 minutes (if there are such events), and maybe an other one will be sent 10 minutes later with new notifications
      Case 1: gmail can grouped emails that concern the same document in a unique mail thread
      Case 2: we make sure we don't have more than 1 email per 10 minutes (the grace period), but grouping is more complicated since it can concern several documents
      Currently we have 1), except that we miss the MessageID field to make gmail group things efficiently

      Thomas Mortagne:
      1 - ou mean for each DocumentUpdatedEvent ?
      s/ou/you/

      Guillaume Delhumeau
      But I thought we had 2
      Thomas: no, for each event. The grace period will make sure you don't receive more than 1 mail per 10 minutes concerning this document
      But you can receive several emails during these 10 minutes if they concern different docs

      Thomas Mortagne
      talking about 1, not 2

      Guillaume Delhumeau
      in 1, we already have a grace period
      I did not mention it for trying to make things easier

      Thomas Mortagne
      so you still have to wait for 10 minutes in both cases ?

      Guillaume Delhumeau
      Yes
      Actually let me rephrase
      1 - One mail per document, but not more than 1 mail per 10 minutes concerning this document
      2 - One mail per 10 minutes, possibly concerning more than 1 document
      Apparently the Watchlist did 1)

      Thomas Mortagne
      if we are waiting anyway I would be for grouping everything that can be grouped (but yes it's probably more work). I guess we could also let the user choose with profile preference

      Guillaume Delhumeau
      Clément Aubin: please confirm I'm not telling something wrong
      *saying

      Thomas Mortagne
      so +1 for 2 but I prefer both 1 and 2 (with 2 being the default)

      Guillaume Delhumeau
      haha, it seems both
      it *means

      Thomas Mortagne
      you have 1 already anyway so if you implement 2 it would be a pitty to ditch it

      Guillaume Delhumeau
      but you prefer 2 if I cannot do both, of course

      Thomas Mortagne
      yes
      but it's not like I was -1 for 1 either
      I'm fine with both, just prefer 2

      Guillaume Delhumeau
      Vincent Massol, Enygma, Marius Dumitru Florea, evalica: please take 2 minutes to look at it

      Thomas Mortagne
      is the grace period configurable already ?

      Guillaume Delhumeau
      it is

      Thomas Mortagne
      ok
      can I put something lower than 10 minutes
      ?

      Guillaume Delhumeau
      minimum is 0

      Thomas Mortagne
      what does 0 will do in practice ? constently execute sql queries to see if something new came up ?

      Eduard Moraru
      Guillaume Delhumeau: you had a look at the realtime watchlist notification, right?

      Guillaume Delhumeau
      Enygma: I looked at your commits about Message-Id and so on (https://github.com/xwiki/xwiki-platform/commit/e38a0db8b93a1b00ced5f35a8e3ab45b5aa2b253)
      I mean https://github.com/xwiki/xwiki-platform/commit/fd93f5cf2cd45444183bdd95f905d463aabae6f1

      Eduard Moraru
      I personally prefer 1, since that was the point of real-time notifications, AFAIU. Also, coupling this with the email client grouping results in a really nice an useful overview of the received emails, allowing to focus on the changed pages. This way, in the email client, if you are watching 5 documents, you will never have more than 5 conversations to read. If you send batches, the best this we could do is to group all those batches under the same conversation ID, so that in the email client you only have 1 conversation, but it might get hard to follow at some point. Also, Gmail automatically breaks a conversation longer than X (around 100, AFAIR) messages.

      Thomas Mortagne
      I would prefer 1 too if you did not had to wait for 10 minutes

      Eduard Moraru
      sure, if 1 also has the grace period / time buffer of around 5 minutes (maybe 10), that would be OK to. This is what we have now, AFAIU.
      yes, 10 mins sounds a bit too much

      Thomas Mortagne
      but since in both cases we have to wait then lets group everything that can be grouped

      Eduard Moraru
      I would even argue that it should be around 30 sec to 1 minute
      and it should be a moving window, based on incoming events
      i.e. high traffic -> longer wait time to get notified of a bigger batch
      small traffic -> faster notification of a smaller batch

      Thomas Mortagne
      personnaly I'm find iwith 2 and I will lower a lot the grace period n my instances
      so be closer to real time

      Eduard Moraru
      what I don't want to see is a notification mail about more than 1 document
      (my preference)
      because then the email client grouping can not be done
      but heck, we could end up with everything configurable (and ages to implement )

      Thomas Mortagne
      yeah maybe, did not tough much about that, in my mind it was about getting a mail and delete it
      I don't think about it like a conversation I need to group like forum notifications

      Guillaume Delhumeau
      ok so at least we need to set the ConversationId (that we don't have now)

      Thomas Mortagne
      but I can understand that after a while it's nicer
      anyway if we keep 1 then I'm cool with that but if we do 2 seems to be configuration does not really add much work (but if it's not doable so be it)

      Vincent Massol
      configurable 1 and 2 is fine with me too
      if we have to choose one, I don't have much of an opinion ATM
      both are fine with me

      Thomas Mortagne
      in any case we have to choose a default
      but if everyone is find with 1 looks like a lot less work
      *fine

      Vincent Massol
      realtime is more 1 probably

      Thomas Mortagne
      we can always do 2 later

      Vincent Massol
      (also 1 cost more in term of perf)
      (more emails)

      Thomas Mortagne
      not sure about that
      more email but less work to group things
      so more work for the mail server, not us

      Vincent Massol
      I was talking scalability of emails
      I'm not sure how well or rather how bad realtime notif is when you have lots of users
      imagine 100K users
      and some activity
      I'm not even sure there's enough time to send all the emails

      Thomas Mortagne
      you usually don't have 100k users listening to everything
      so you almost never send that many emails

      Enygma
      well, during the night they will eventually get the emails

      Thomas Mortagne
      and the difference between 1 and 2 is probably something like sendig 1 email instead of a few most of the time
      (per user listening to those changes)

      Vincent Massol
      So one question could be: how many users using realtime do we need to make the sending not "realtime", i.e. taking longer than, say, 10 minutes
      (knowing that we have a wait perdio before sending each mail, by default it's 8 seconds I think)

      Thomas Mortagne
      what wait period are you talking about ?

      Vincent Massol
      yes it's 8s (just checked)

      Thomas Mortagne
      the mail module wait for 8s between each mail ?

      Vincent Massol
      yes
      "The delay to wait between each mail being sent, in milliseconds (defaults to 8 seconds)"

      Thomas Mortagne
      why ?

      Vincent Massol
      to not be considered a spammer
      the value wasn't chosen randomly
      I read the doc on several mail server to find the value that would not consider XWiki a spammer
      it's configurable ofc

      Thomas Mortagne
      so this is only when there is several mails, right ?
      if you send a mail it's sent right away ?

      Vincent Massol
      yes

      Thomas Mortagne
      so it's not "before sending each mail"

      Vincent Massol
      (when thre are several to be sent in the queue)
      So one question could be: how many users using realtime do we need to make the sending not "realtime", i.e. taking longer than, say, 10 minutes

      (knowing that we have a wait perdio before sending each mail, by default it's 8 seconds I think)
      ^^^so there'll be several in the queue

      Thomas Mortagne
      the main criteria is not really teh number of users I think, it all depends what those user are watchling, no ?
      you are not going to send all changes to all users all the time, right ?
      S
      Vincent Massol
      it should depend on users, activity in the wiki and indeed what users are watching

      Thomas Mortagne
      in any case the difference between 1 and 2 in number of email is probably not going to be very big most of the time unless the edit activity is very high and wide

      Vincent Massol
      yes, my question was not so much for 1 and 2 but more in general

      Thomas Mortagne
      not sure what is you question then ? should we do real time notifications ?
      IMO as long as it does not take much resources mail will arrive when they arrive

      Vincent Massol
      so 10*60/8=75, so we can serve 80 emails per 10 minutes.
      so if we have 80 users all folllowing one doc, the last reatime email will arrive after 10 minutes
      (at best)
      (then ofc if you're using this internally with your own mail server you can reduce the wait time to 0s)
      (and then it's about how many emails we can prepare and send per seconds)
      (I don't know this figure)
      anyway, back to clover work, I've found a way to compare the reports I think

      Thomas Mortagne
      Guillaume Delhumeau: so looks like 1 with proper conversation ids

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                gdelhumeau Guillaume Delhumeau
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: