Before Nested Spaces, we had a constant hierarchy level in Document or Space References: wiki, space and page.
With Nested Spaces, this hierarchy is variable and the default implementation of the compare method performed in EntityReference.compare is no longer appropriate, since that generic implementation works by comparing depth and only for references of equal depth and of the same type is the name compared.
A problematic example is:
["A.B", "X.Y", "A.B.C"] is considered sorted (using only length) instead of the expected result:
["A.B", "A.B.C", "X.Y"]
The problem is even more evident when it comes to wikis:
["a:A.B", "b:X.Y", "a:A.D", "b:A.B.B", "a:A.B.C"] is considered sorted instead of the expected:
["a:A.B", "a:A.B.C", "a:A.D", "b:A.B.B", "b:X.Y"]
The conclusion is that we need specific implementations for DocumentReference and SpaceReference that understand what it is they are sorting. Some improvements to the generic EntityReference implementation could be done, for example when comparing a SpaceReference to a DocumentReference, since it is not very useful to compare them only by length.
Workaround: Until this is properly implemented, if you know you are sorting a list/array of DocumentReferences only or of SpaceReferences only, then you can have a custom comparator that compares the serialized string references to produce the expected result.