IMO it shouldn't list 'Blog'
I assume you are referring to the Spaces gadget, right? If so, I agree.
Now the problem is how to configure this. Right now we only have the notion of hidden documents. Of course we could modify the Blog app to create blog posts as hidden but that's not good because they wouldn't appear in search for example.
Thus, IMO we need another way. This could be a new XObject attached to a space WebPreferences page and the Spaces macro would not display spaces that have this XObject.
it shouldn't list 'XWiki' because I actually can't access it and I shouldn't care about it since mostly it contains hidden pages.
Not quite. It contains all XWiki users which is why it's not hidden. The solution I've listed a few lines above would solve this too.
IMO allowing the user to configure the list of spaces to see/not see is another feature which can be implemented on top of my solution. We would still need a default solution and we need new extension to define how they consider their spaces by default. The XObject attached to WebPreferences allow for this.