[bt-devel] Fwd: Re: clucene crash when searching

[Joachim Ansorg]

I don't remember exactly why we switched but here's what I think, in random 
order :)

We always had troubles with the Sword search engines. I don't know how the 
current implementation is, though.

-It's  slow. Really slow. Using modules with many tags (like the KJV) the 
search has taken a long time to finish. Especially on non high-end computers 
it takes a long time. More than 5  seconds is not an acceptable response time. 
Often it had been worse, I think. And that's just for seaches in one module at 
a time.
-No complex searches are possible. and/or/grouped combinations are not easily 
done with the sword engine.
-We had no control where the index files are stored (using the clucene 
implementation of Sword).
-clucene was sometimes available in the sword lib, sometimes not. That 
depended on how the user configured and installed his libsword.
-It's a mess to work with different types of search engines which all have 
different feature sets.

So using our own search engine where we have the control about query 
expressions, index storage, encoding, indexing options, etc. has been a good 
solution. 
And Sword's development was quite slow at that time so we had to move forward 
with our own solution because we never wanted to work with the Sword SVM head 
for the next release.

The problem is that CLucene is almost unmaintained and crashes on certain 
kinds of systems (got reports about crashes on BSD for example). Java Lucene 
is much, much better. We've hoped that CLucene developes like JLucene, but 
sadly that didn't work that way...

Hopefully that helps a bit :)
Joachim

> BTW, speaking of the search capability: I still think we should support
> the default Sword search engines (though not the clucene one). One
> reason is that the clucene search is unreliable - it's only as reliable
> as the indexing and syntax engines are, and we have seen that they are
> not perfect. Bible is different from other texts, as I said earlier,
> because it may be necessary to find words and sentences as they are
> literally. Clucene doesn't allow that, at least not without extra
> customizing. The second reason is that clucene doesn't support other
> languages. We saw that with the request for French support. The third
> one is that people working with Bible may need as reliable results as
> possible and no one search engine is perfect. Several engines would give
> extra reliability for those who want to double check.
>
> Therefore I would like to hear why you consider supporting the other
> engines a bad thing. What exactly are the problems? I might come up with
> solutions, as you probably have seen with some other features :)