[bt-devel] BibleTime Search

[Lee Carpenter]

Joachim Ansorg wrote:
> All your additions worked out of the box!
> And they are a great addition to BibleTime!
> Thanks a lot for your help to get this important piece done! This is so cool.
> I really like what you did!

Thanks :)

> Here are some things I noticed:
> 	-The index is rather large. I think JSword and Sword had indexes less than 
> 10megs per module.
> I played a bit with the indexing. By using Field::UnStored instead of ::Text 
> for the content and parameters we save about 50% of the disc space.
> I also noticed that the index dir contained a lot of files. I set 
> minMergeFactor to 500 and the index is now 11megs instead of 28 megs (or of 
> 56megs if we use Field::Text).
>>From what I've read the optimize call should merge all those many files into a 
> few. It seems that optimize is not working for me. Strange.
> Does optimize work on your side?

Yes, I noticed the huge index too, especially for KJV that has all the 
footnotes, headings, strongs and morphs.  This pass was my "just get it 
working pass" so I haven't looked in depth at the tuning one could do 
with CLucene.  Your comments and trials have given me a big headstart on 
it though! Thanks!  I'll look into to these.

> 	-The search result is not in the biblical order.
> I noticed a search() method which takes a Sort object. But I had no success to 
> get it working. It crashed for me all the time, as I tried to add a new field 
> "index" which stores the key position in the module. I'm suign 0.9.10.

I noticed that too.  I wondered if it was in some sort of "score" order. 
    Again, I hadn't looked at the CLucene options enought to tell.  It 
might be useful to have several sorting options available here...

> 	-The search result highlighting highlights the AND of clucene, too.

Yeah, noticed that too :)

> 	-I also noticed that search() can take a Filter, too. Perhaps we could use 
> that to limit a search to a search scope?

Hmm, hadn't thought about that.  Search scope should work already (other 
than "last result"), but that might be a cleaner way to do it...

> You don't have to do this all by yourself. Please tell me what to do, if you 
> don't want to all of the remaining things you have on your list.

Thanks.  I enjoy working on this, but it is hard to find time some days. 
  I work best if I have "blocks of time" greater than 2 hours or so. 
That can be challenging with a full time job, a 3 year old running 
around and another on the way (February) :)  I just hope my pace isn't 
too slow to be useful to you guys...  That said, I'm not too picky.  So 
if you want to pick up some piece of my TODO list and you have the time, 
just let me know and I'll work on the other pieces.  If not, I don't 
mind finishing them all if you can handle my pace :)

> I'm sorry for the horrid code of BibleTime. The backend has been there for 
> years now and was written as I still was learning how to code. We're open for 
> suggestions how to refactor that :)

:)  I wouldn't worry about it.  I haven't seen anything in there that 
scared me :)  I don't know what your experience is with "professional" 
codebases, but I've had a hand in 7 or so different codebases with 3 
different companies in my career.  Let's just say that "professional" 
only means that people were paid to do it and usually does not reflect 
the quality of the code ;)  The codebase I work in now is by far the 
best I've used and it still has pieces you don't really want to venture 
into...  But since you offered, if I see anything that I think could 
benefit from some refactoring I'll mention it.

> May our Lord bless you for the work you do,
> Joachim
Thank you!  He has!