I experienced the issue when using a listener on user profile creation (DocumentCreatedEvent) which adds the user to some groups based on information in the user profile. In my particular case the user is created by an LDAP Authentication.
The code that sets the initial groups of an user on user creation ( https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/XWiki.java#L3504 ) also calls the GroupService#addUserToGroup, which is supposed to add the new user membership to the groups cache. However, this function is very badly written, assumes that if there is no entry in the cache for a given user, it initializes the cache with the empty list, when it should actually check the user groups in the database - see the code here https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/user/impl/xwiki/XWikiGroupServiceImpl.java#L221. In my case this is obviously incorrect, the user group cache is empty, but the user has some group memberships which were set in the listener.
This caused the list of user groups to be set to XWikiAllGroup in the cache, and, since the cache was set, no database reading of the list of groups for the user was done to fetch the membership that I set in the listener.