XWiki Platform
  1. XWiki Platform
  2. XWIKI-9294

Display the indexing progress in the Solr administration section

    Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.1-rc-1
    • Fix Version/s: 5.1
    • Component/s: Search - Solr
    • Labels:
    • Difficulty:
      Unknown
    • Similar issues:

      Activity

      Hide
      Vincent Massol added a comment -

      FTR JV introduced the feature in the Lucene search in https://github.com/xwiki/xwiki-enterprise/commit/a74d1674fc26a6296d70d6b437a22dfb1f620cd5

      At that time, the condition to display the queue size was that the queue had at least 2 elements:

      +        #set ($queuesize = $xwiki.lucene.getQueueSize())
      +        #if ($queuesize > 2)
      +            #set($info1 = "Lucene is currently building its index, ${queuesize} documents in queue.")
      +        #else
      +            #set($info1 = "")
      +        #end
      

      Apparently the check was modified by Thomas in SVN rev 28693 (from 2 to 0) when moving the Search from the Panels module to the Search module.

      Show
      Vincent Massol added a comment - FTR JV introduced the feature in the Lucene search in https://github.com/xwiki/xwiki-enterprise/commit/a74d1674fc26a6296d70d6b437a22dfb1f620cd5 At that time, the condition to display the queue size was that the queue had at least 2 elements: + #set ($queuesize = $xwiki.lucene.getQueueSize()) + #if ($queuesize > 2) + #set($info1 = "Lucene is currently building its index, ${queuesize} documents in queue.") + #else + #set($info1 = "") + #end Apparently the check was modified by Thomas in SVN rev 28693 (from 2 to 0) when moving the Search from the Panels module to the Search module.
      Hide
      Thomas Mortagne added a comment -

      Note that the Lucene queue is for the whole farm too.

      Show
      Thomas Mortagne added a comment - Note that the Lucene queue is for the whole farm too.
      Hide
      Ludovic Dubost added a comment -

      Hi,

      It is regularly a need to have more information about what is in the index. We even developped a scheduler job to look at the latest changed documents to verify if the index is up to date which we used for cluster mode (which could lead to some documents missing to be indexed).

      Once we installed the lucene navigation tool to look at the index.

      The most important is to know how many documents there are, to see if changing documents actually does make the number move. However it would be very valuable to be able to navigate the index to understand if the search is not finding a document because it's missing from the index or if the search is badly looking it up.

      Show
      Ludovic Dubost added a comment - Hi, It is regularly a need to have more information about what is in the index. We even developped a scheduler job to look at the latest changed documents to verify if the index is up to date which we used for cluster mode (which could lead to some documents missing to be indexed). Once we installed the lucene navigation tool to look at the index. The most important is to know how many documents there are, to see if changing documents actually does make the number move. However it would be very valuable to be able to navigate the index to understand if the search is not finding a document because it's missing from the index or if the search is badly looking it up.
      Hide
      Ludovic Dubost added a comment -

      Once thing that was very important with lucene was incremental reindexing (looking each doc and only reindex if the document is missing). This was developped for Curriki which had a very large index and full reindexing was too long.

      It would also be interesting to know how many and which documents led to an indexing error.

      Show
      Ludovic Dubost added a comment - Once thing that was very important with lucene was incremental reindexing (looking each doc and only reindex if the document is missing). This was developped for Curriki which had a very large index and full reindexing was too long. It would also be interesting to know how many and which documents led to an indexing error.
      Hide
      Marius Dumitru Florea added a comment -

      Thanks Ludovic for the feedback. We need to release 5.1 so we'll take it into account when we iterate over the administration section in the next version. To summarize:

      • display the total number of indexed documents (per wiki?)
      • display documents that failed to be indexed
      • ideally, display a live table with the indexed documents
      Show
      Marius Dumitru Florea added a comment - Thanks Ludovic for the feedback. We need to release 5.1 so we'll take it into account when we iterate over the administration section in the next version. To summarize: display the total number of indexed documents (per wiki?) display documents that failed to be indexed ideally, display a live table with the indexed documents

        People

        • Assignee:
          Marius Dumitru Florea
          Reporter:
          Marius Dumitru Florea
        • Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:
            Date of First Response: