Some improvement of the solr search was done in issue
XWIKI-12043 in 7.1 and announces an improvement of 30% in speed. However, that issue seems to be related to only the html cleaner (it's not very clear what are the improvements that were done as there is no commit for that issue).
However, an analysis shows that the number of html macros used could be decreased, to improve performance, and cleaning could be skipped alltogether for these macros to improve performance. Also, a modification is possible in the way facets are created and used, which would also improve performance (but this change would affect the backwards compatibility).
The results that were obtained with a search patched with the attached patch are somewhere around four times faster than before (which would mean 75% faster, I guess it also depends on the amount of data, this amount was significant). It would be interesting to study this patch and see if anything can be taken from it to go one step further than
Credits for the patch go to Denis Gervalle.