[jsword-devel] Fwd: [jsword] JS-98: IndexManager.needsReindexing(Book book); as the API to detect if per-book index need to be recreated (#74)

Chris Burrell chris at burrell.me.uk
Mon Apr 7 01:59:34 MST 2014 

Hi Sijo

STEP has its own datasources (not sword modules), in the form of Lucene
indexes. I'm not sure if the functionality is there already, but it would
be a nice plus I were able to version those indexes together with the Sword
modules. (But it's a plus, and so shouldn't drive design.)

Chris



On 6 April 2014 22:17, Sijo Cherian <sijo.cherian at gmail.com> wrote:

> Thanks Chris.
>
> Is the only usecase for "external Lucene indexes" to install a
> downloadable index?
>
> I am assuming any "external Lucene indexes" usage is going through
> IndexManager interface (in our default case LuceneIndexManager).
> In that case I will add version management through IndexManager api.
>
> So the Installed index versions will go in the folder returned by
> LuceneIndexManager.getStorageArea(Book book)
>
>
> cheers back
> sijo
>
>
> On Sat, Apr 5, 2014 at 5:24 PM, Chris Burrell <chris at burrell.me.uk> wrote:
>
>> Sorry about the radio-silence. I agree with the thread so far. STEP also
>> uses that folder to deposit its own set of Lucene indexes. So that would be
>> a nice central place.
>> '
>> 'twould be a nice feature if we could register external Lucene indexes
>> with JSword such that an external app could use Sijo's work. In other
>> words, for other indexes, external to JSword, it would be nice to be able
>> to ask whether indexes could be upgraded, etc.
>>
>> Cheers,
>> Chris
>>
>>
>>
>> On 2 April 2014 23:34, DM Smith <dmsmith at crosswire.org> wrote:
>>
>>>
>>> On Apr 2, 2014, at 5:23 PM, Sijo Cherian <sijo.cherian at gmail.com> wrote:
>>>
>>> I will appreciate feedback or concern from jsword users (And,BD..) on
>>> having a property file to store current index version (currently indexed
>>> default version + per-book's index version, if different).
>>>
>>> After that, any recommendation on suitable location for this file (I am
>>> thinking {IndexFolder}/JSword/lucene/js.index.metadata.prop file). I am
>>> not sure if all platforms have same folder structure (like
>>> JSword/lucene)
>>>
>>>
>>> I think it should be in the folder that holds the lucene index files.
>>> I'm not sure if AndBible has done anything special to the code that locates
>>> the lucene indexes, but the JSword code that determines the state of the
>>> index presumes the location. It differs by platform, but that is
>>> transparent within JSword. I'm going to guess that AndBible has handled
>>> pathing so that it also is transparent. So I think that if you can put it
>>> there all front-ends should work with it.
>>>
>>> note: In code comments, I am referring to this file as
>>> InstalledIndex.prop.
>>>
>>>
>>> Regarding the writing and retrieval of values it is up to you. You might
>>> be able to add the API to IndexManager.
>>>
>>> It seems that IndexManager should have a IndexStatus getIndexStatus().
>>> (search for setIndexStatus and you'll see why). Ultimately a user asks
>>> getIndexStatus from Book, which delegates the question to BookMetaData. The
>>> status INVALID is defined as the index is no longer valid. Nothing sets
>>> this. Nothing really uses it either.
>>>
>>>
>>> Many thanks,
>>>
>>> sijo
>>>
>>>
>>>
>>> On Sun, Mar 30, 2014 at 4:27 PM, DM Smith <dmsmith at crosswire.org> wrote:
>>>
>>>> Sijo asked this via github. Thought it'd be good to get feedback here.
>>>>
>>>> -- DM
>>>>
>>>> Begin forwarded message:
>>>>
>>>> *From: *Sijo Cherian <notifications at github.com>
>>>> *Subject: **[jsword] JS-98: IndexManager.needsReindexing(Book book);
>>>> as the API to detect if per-book index need to be recreated (#74)*
>>>> *Date: *March 29, 2014 at 7:31:52 PM EDT
>>>> *To: *crosswire/jsword <jsword at noreply.github.com>
>>>> *Reply-To: *crosswire/jsword <
>>>> reply+i-30455246-2897689b705296a200fa088d2af792bed6bc6a4a-1645068 at reply.github.com
>>>> >
>>>>
>>>> Following is the upgrade criteria implemented by comparing
>>>> IndexMetadata from prop files: needsReindexing returns true, if
>>>> Latest.Index.Version.xxx > Installed.Index.Version.xxx OR if {index folder}
>>>> is not found at all.
>>>>
>>>>    - For now uses two prop file: 1. IndexMetadata.prop file (can be
>>>>    updated with each release of JSword) 2. InstalledIndex.prop file for
>>>>    "Installed.Index.Version.xx" values that should stay persistent on clients
>>>>    computer between upgrades
>>>>    - Support for PerBookVersion property: Since index creation api is
>>>>    per Book based, added support for (optional) PerBookVersion properties to
>>>>    allow selective upgrade of one book's index, if needed.
>>>>    - This pull request, still assumes index version as 1.2, so should
>>>>    not break anything
>>>>
>>>> Notes: Looked into storing version values in the lucene index using
>>>> FlexibleIndexing in lucene 4.0, but that seem very dependent on lucene api.
>>>> Prop file gives human readable configuration and easy to debug.
>>>>
>>>> ToDo/Questions:
>>>> 1. Managing the add/update of values in InstalledIndex.prop
>>>> programmatically
>>>> 2. Need recommendation on suitable location for InstalledIndex.prop
>>>> file (perhaps as {IndexFolder}/JSword/lucene/js.index.metadata.prop file)
>>>> 3. Need to add reindexAllBooksIfNeeded() API, that folks (AndBible, BD,
>>>> Alkitab and?) can use to upgrade all index as bulk, for the installed books
>>>> , only then the DefaultInstalledVersion can be updated.
>>>> o.c.j.bridge.BookIndexer.java shows a sample API in the comments.
>>>> 4. How do the clients want the DownloadIndex option to work with index
>>>> versioning?
>>>> ------------------------------
>>>> You can merge this Pull Request by running
>>>>
>>>>   git pull https://github.com/sijocherian/jsword master
>>>>
>>>> Or view, comment on, or merge it at:
>>>>
>>>>   https://github.com/crosswire/jsword/pull/74
>>>> Commit Summary
>>>>
>>>>    - JS-98 : mechanism to detect if index need to be recreated by
>>>>    comparing IndexMetadata from prop file for now Started partially with the
>>>>    patch from Tonny Kohar dated 15/May/09
>>>>    - - Added InstalledIndex.prop file (location of this file TBD)
>>>>
>>>> File Changes
>>>>
>>>>    - *M* src/main/java/org/crosswire/jsword/bridge/BookIndexer.java<https://github.com/crosswire/jsword/pull/74/files#diff-0>(20)
>>>>    - *M* src/main/java/org/crosswire/jsword/index/IndexManager.java<https://github.com/crosswire/jsword/pull/74/files#diff-1>(16)
>>>>    - *M*
>>>>    src/main/java/org/crosswire/jsword/index/lucene/IndexMetadata.java<https://github.com/crosswire/jsword/pull/74/files#diff-2>(29)
>>>>    - *A*
>>>>    src/main/java/org/crosswire/jsword/index/lucene/InstalledIndex.java<https://github.com/crosswire/jsword/pull/74/files#diff-3>(67)
>>>>    - *M*
>>>>    src/main/java/org/crosswire/jsword/index/lucene/LuceneIndexManager.java<https://github.com/crosswire/jsword/pull/74/files#diff-4>(53)
>>>>    - *M* src/main/resources/IndexMetadata.properties<https://github.com/crosswire/jsword/pull/74/files#diff-5>(18)
>>>>    - *A* src/main/resources/InstalledIndex.properties<https://github.com/crosswire/jsword/pull/74/files#diff-6>(39)
>>>>    - *A*
>>>>    src/test/java/org/crosswire/jsword/index/lucene/LuceneIndexManagerTest.java<https://github.com/crosswire/jsword/pull/74/files#diff-7>(131)
>>>>
>>>> Patch Links:
>>>>
>>>>    - https://github.com/crosswire/jsword/pull/74.patch
>>>>    - https://github.com/crosswire/jsword/pull/74.diff
>>>>
>>>> --
>>>> Reply to this email directly or view it on GitHub<https://github.com/crosswire/jsword/pull/74>
>>>> .
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> jsword-devel mailing list
>>>> jsword-devel at crosswire.org
>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Sijo
>>> _______________________________________________
>>> jsword-devel mailing list
>>> jsword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>
>>>
>>>
>>> _______________________________________________
>>> jsword-devel mailing list
>>> jsword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>
>>>
>>
>> _______________________________________________
>> jsword-devel mailing list
>> jsword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>
>>
>
>
> --
> Regards,
> Sijo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20140407/c2d0f95e/attachment-0001.html>

More information about the jsword-devel mailing list