The current refactoring module defines job groups base on the wiki of the affected entity. If entities from multiple wikis are affected, a farm group will be used.
At any time, a maximum of one job will be run per job group.
This limitation becomes highly problematic when we start using interactive jobs (with question/answers) because they have the big potential of blocking the entire group until the user manages to answer the question. This problem exists with non-interactive jobs as well, in case the job simply fails to finish for whatever reason (endless loop, deadlock, etc.).
To improve this, Marius Dumitru Florea is suggesting that we assign the groups based on the path of the affected entity. Example:
- jobs on /A/B/C can be run in parallel with jobs on /X/Y/Z...
- however, jobs on /A/B/C will wait for any already running jobs on /A (path prefix)
This has the advantage that it decongestionizes the jobs that are not supposed to interact with eachother, though 3 issues still remain:
- if a job on /A/B/C is already started, a new job on /A will not be blocked (it works only the other way around, at least in the current implementation)
- what happens when multiple entities are affected, how will the group be determined then? Most probably the highest denominator will be chosen, so still a high chance of blocking.
- it is still very easy to block everything, either intentionally (trolling/malicious user) or by delay in the user's answer to a question in the UI...
Jobs would be free to use whatever group they want, or even a random group each time (to go fully parallel), but we need to come up with a coherent solution for the refactoring module's jobs at least and then see if the job module itself needs changes as well.
IMO, we should try to think of how the chosen solution will function in a large and busy wiki where waiting for stuff like CRUD operations is a challenging thing to ask of users. (I mean not waiting for the operation the user does, but waiting for others to finish the operations that they have started).