XWiki Lucene Plugin

I came across this issue on a production site. It was using xwiki-plugin-lucene-1.12.

org.apache.lucene.search.IndexSearcher

from lucene-core-2.9.1.jar has this comment for the close() method

/**

• Note that the underlying IndexReader is not closed, if
• IndexSearcher was constructed with IndexSearcher(IndexReader r).
• If the IndexReader was supplied implicitly by specifying a directory, then
• the IndexReader gets closed.
  */

The issue is that the underlying reader is not closed, and on Solaris at least, dtrace can be used to see that several cfs files are opened but not closed before unlinking. That is the cause of the file descriptor leak.

A simple solution is to change the way

com.xpn.xwiki.plugin.lucene.LucenePlugin

public Searcher[] createSearchers(String indexDirs, XWikiContext context)

method adds to searchersList i.e. collapse

IndexReader reader = IndexReader.open(dirs[i], true);
searchersList.add(new IndexSearcher(reader));

to

searchersList.add(new IndexSearcher(dirs[i], true));

That way the IndexReaders are closed and no more fd leaks occur.

Thanks a lot Chris, i'm trying that right now. I will ping you here when a 1.15-SNAPSHOT is available with the fix so that you can validate it for your use case.

Are you sure it's enough ?

I just looked at the Lucene source and i can see that it seems to be the same thing actually:

public IndexSearcher(String path, boolean readOnly) throws CorruptIndexException, IOException {
    this(IndexReader.open(path, readOnly), true);
  }

Ha no there is a difference actually but it simply to use

searchersList.add(new IndexSearcher(reader, true));

instead of

searchersList.add(new IndexSearcher(reader));

because it's false by default

Thanks for the catch anyway it's nicer the way you did it.

Thomas, are you confusing the readOnly boolean with the close reader boolean, or missing that IndexSearcher(reader, true) is private?

The fix is in supplying the directory to the IndexSearcher constructor so that it then calls the private constructor with closeReader set true;

public IndexSearcher(String path, boolean readOnly) throws CorruptIndexException, IOException { this(IndexReader.open(path, readOnly), true); }

instead of using

public IndexSearcher(IndexReader r) { this(r, false); }

As i said i understood the difference, I just did not seen that IndexSearcher(IndexReader r, boolean closeReader) was private (I did not used it anyway).

BTW you can now download Lucene plugin with the fix a just committed on http://maven.xwiki.org/snapshots/com/xpn/xwiki/platform/plugins/xwiki-plugin-lucene/1.15-SNAPSHOT/xwiki-plugin-lucene-1.15-20100202.172542-12.jar

I can confirm that the lsof command now only shows one (1) open lucene index file, when rebuilding xwiki with lucene plugin 1.15-SNAPSHOT.

java 30943 serasset 15r REG 14,2 469927 13274923 /Users/serasset/dev/xwiki/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/work/Catalina/localhost/xwiki/lucene/_s.cfs

This index file does change when time goes on, and older index file do not appear in lsof anymore.

java 30943 serasset 15r REG 14,2 469981 13275082 /Users/serasset/dev/xwiki/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/work/Catalina/localhost/xwiki/lucene/_u.cfs

This shows that the fix works in a development environment. I'm eager to install xwiki 2.2 in a deployment environment to get rid of this bug that obliged me to relaunch tomcat regularly.

Env: Mac OS 10.5, xem-web-debug compiled with maven, with mysql db.

Great ! I hope it really fix the issue entirely.

Thomas,

I applied xwiki-plugin-lucene-1.15-20100202.172542-12.jar to a test system which was also exhibiting the leak and the fix appears to be good.

Thanks for the prompt action.

Chris