Sometimes rights changes don't apply without a server restart

I noticed this issue on multiple XWiki instances, of which one for sure is 5.4.5, but I don't know how to reproduce it as I don't know when it happens.

On the 5.4.5 the situation was as follows: a document was not visible to guest, and we looked at all the rights on the page, space and wiki and could not find anything that would explain this.
Actually, the rights settings were as follows (in case it matters): on the wiki the guest had view rights (and register) along with xwiki all group. On the space, there were some rights settings for other groups, but none of these groups had settings for the "view" right. On the page there were no rights objects.

Then, out of solutions, we restarted the wiki.
Everything was fine, guest could access the page with no other modification.

I checked the history of the document itself and it seems that, 2 days before and a couple of versions of the document before, view right was denied for the guest (one object of type XWikiRights added) and then, 1 minute later, this object was removed (so rights should have gone back to guest). These changes are very, very probably legitimate user changes, but the rights did not go back until the server restart.

I saw this issue 2-3-4 times before, but since I couldn't reproduce it it was hard to report.