[sword-devel] CLucene

[DM Smith]

On Aug 14, 2006, at 2:42 AM, Joachim Ansorg wrote:

> DM,
> thank you for your comments on that topic.
>
> You're right, that updates will break the native CLucene order. But  
> since
> updates are not possible in Sword at the moment using CLucene to  
> sort the
> results should work fine.
>
> I can't think of a situation where updates could be used for a  
> module index,
> though.

Joachim,
I can think of a couple reasonable scenarios, and some far fetched  
ones, where adding index update capability might be needed.

How about personal commentary? I can imagine a situation where  
someone's comments become very extensive and would benefit from  
boolean queries that Lucene provides.

How about if we added the ability to create the index at module  
construction time? One of the problems with constructing an index on  
the client machine is that it may be slow and may consume much  
resources. By building an index offline and downloading it we can  
avoid some of these problems. While a module input should be ordered  
properly, the module construction mechanism does not require it and  
will re-order out of order verses. Also, it is possible to correct a  
small number of verses in a module without rebuilding the module from  
scratch.

While no user interface currently provides module editing capability  
beyond personal commentary, I can see value in adding such a  
capability to a Sword app. (E.g. create an OSIS editor.) This could  
be used for correcting and creating module bug reports, or for  
creating modules from scratch.

If the client application uses the SWORD api for sorting an answer  
set then it is an encapsulated implementation detail and sorting  
using Lucene document order would be a fine mechanism until these  
other abilities (or others) are added.
DM


>
> Joachim
>
>> Joachim,
>>
>> I could be wrong, but I'm not sure that CLucene's sort would product
>> the right results.
>> IIRC, we store the OSIS ids and these are what need to be sorted into
>> Biblical Order.
>>
>> If it were true that the verses (aka lucene documents) were added in
>> order then one could use the ordering of the document ids (Java
>> Lucene's default sort) to produce the correct order, but if there are
>> any out of order adds then that would not work.
>>
>> Currently the algorithm for building the index will create an index
>> with document order equal to canonical order. However, write support
>> has not been added to an index yet. That is if the application re-
>> writes a verse in the index, the Lucene index is not updated. When
>> update support is added to the index, then document order sort
>> becomes meaningless.
>>
>> It is possible to sort on a field, but even then the ordering of OSIS
>> ids would not return the right results. We want Gen to sort earlier
>> than Exo.
>>
>> DM
>>
>> On Aug 13, 2006, at 7:04 PM, Joachim Ansorg wrote:
>>> The current clucenen has support for the native order, it just
>>> needs to be
>>> patched into sword.
>>>
>>> Joachim
>>>
>>>> The ListKey container which is returned from SWModule::search has a
>>>> newly added .sort() method.
>>>>
>>>> Eeli Kaikkonen wrote:
>>>>> On Sun, 6 Aug 2006, Eeli Kaikkonen wrote:
>>>>>> I tried to compile Sword for some time ago on Debian unstable.
>>>>>> CLucene
>>>>>> had some problem then. Now I tried again (after upgrading
>>>>>> Debian) and
>>>>>> compilation worked for some reason, maybe CLucene had been  
>>>>>> updated.
>>>>>>
>>>>>>
>>>>>> Now the problem is that I don't know how to use clucene with
>>>>>> sword. Any
>>>>>> pointers? I just like to test it, preferably with some simple
>>>>>> command
>>>>>> like tool like diatheke.
>>>>>
>>>>> I managed to compile BibleTime1.6beta3 with clucene support. I  
>>>>> made
>>>>> indexes with that and copied them to sword module directories. I
>>>>> made
>>>>> necessary changes to diatheke and now it supports clucene search.
>>>>>
>>>>> /work/sword/bin/diatheke -b KJV -s lucene  -k "what*"
>>>>>
>>>>> Verses containing "what*"-- I Samuel 20:4 ; Genesis 2:19 ; Romans
>>>>> 2:29
>>>>> ; [...] Revelation of John 3:3 -- 1036 matches total (KJV)
>>>>>
>>>>> It took about 2 seconds when the index was in physical memory disk
>>>>> cache. Great. Downside is that returned verse keys are not
>>>>> alpabetically
>>>>> arranged.
>>>>>
>>>>>   Yours,
>>>>> 	Eeli Kaikkonen (Mr.), Oulu, Finland
>>>>> 	e-mail: eekaikko at mailx.studentx.oulux.fix (with no x)
>>>>>
>>>>> _______________________________________________
>>>>> sword-devel mailing list: sword-devel at crosswire.org
>>>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>>>> Instructions to unsubscribe/change your settings at above page
>>>>
>>>> _______________________________________________
>>>> sword-devel mailing list: sword-devel at crosswire.org
>>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>>> Instructions to unsubscribe/change your settings at above page
>>>
>>> --
>>> <>< Re: deemed
>>> www.bibletime.info
>>>
>>> _______________________________________________
>>> sword-devel mailing list: sword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>> Instructions to unsubscribe/change your settings at above page
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>
> -- 
> <>< Re: deemed
> www.bibletime.info
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page