[sword-devel] lucene indexing failing on some modules

[DM Smith]

Karl Kleinpaste wrote:
> I have found that indexing is not working on some modules, including
> some of my home-grown ones, but also a few Crosswire-distributed ones.
> Case in point, BosworthToller.  This was first noticed from integrated
> support in GS, but use of mkfastmod on its own shows similar problems.
> In 15min wall clock time, mkfastmod grew to 126M and used 13:20 CPU
> without completing, or generating anything in the lucene subdir in the
> module's tree.


The code is in .../modules/swmodule.cpp.

The way indexing works is that:
It creates the location that the index should write to.
Then it uses a RAMDirectory to write the index to.
Then there is an ugly loop that iterates over the entries in a module, 
getting their keys and contents, inverting them into the index.
Perhaps of interest is that the first MAX_CONV_SIZE (i.e. 2047) of the 
content is used.
Once it is done, it opens an index on disk and writes the RAMDirectory 
to it.

It would be a simple matter to add a diagnostic to the loop to have it 
spit out the key.

Just a thought:
I would think that if one could run mod2imp successfully on the module 
that the indexer would work.

In Him,
    DM