<div dir="ltr">Hi Sijo<div><br></div><div>Thanks for all the good work. I'm not going to be able to test this much until early June I think, as we're hoping to have a big release going out the door with STEP. Upgrading the Lucene index for STEP will be non-trivial as we use it internally for all our other data sources. Only had a very very quick glance, but looks good. Thanks!</div>
<div><br></div><div><br></div><div>Chris</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 11 May 2014 05:09, Sijo Cherian <span dir="ltr"><<a href="mailto:sijo.cherian@gmail.com" target="_blank">sijo.cherian@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Using this codebase, I did some tests using: <br>- With version = LUCENE_31 : Tested with French FreSegond & Chinese ChiNCVs bible. Some differences found with existing index. So index won't be back compatible<br>
- With version = LUCENE_48, QueryParser yielded same output as LUCENE_31 for French/Chinese tests<br><br>After a transition phase, I am thinking we will switch this version to late and give a new version to JSword-indexing-schema as well. The the downstream projects can prompt user to rebuild/download index of the one book (that is being search).<br>
<br></div>/sijo<br><div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">On Sun, May 11, 2014 at 12:05 AM, Sijo Cherian <span dir="ltr"><<a href="mailto:sijo.cherian@gmail.com" target="_blank">sijo.cherian@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div>This new pull request is intended as a new branch:<br><a href="https://github.com/crosswire/jsword/pull/82" target="_blank">https://github.com/crosswire/jsword/pull/82</a><br>
<br>This pull-request includes code changes to use the new index/query api in Lucene 4.8.0.<br>
(I called jsword release.version = 2.0.1-luceneupgrade-alpha)<br>It uses LUCENE_30 version (set in IndexMetadata.LUCENE_IDXVERSION_FOR_INDEXING) in IndexWriter/QueryParser for compatibility.<br>After a transition phase, I am thinking we will switch this version to latest (to use newer features, less RAM usage in newer index format).<br>
<br>As far as I could test, it seem to be back compatible to existing index when using LUCENE_30. We need folks to test this for european/asian language bibles using existing index (since English is using SimpleLuceneAnalyzer with no stemming etc it is unlikely to have issues in English bible index).<br>
<br>All feedbacks/testing efforts are very much appreciated.<br>/sijo<br><br><br></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Thu, May 8, 2014 at 12:36 AM, Sijo Cherian <span dir="ltr"><<a href="mailto:sijo.cherian@gmail.com" target="_blank">sijo.cherian@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div>I started with changing jsword usage of Lucene api to latest in 4.x (I think this is the easier piece & I am close to 75%).<div>
<br></div><div>Issue is with existing index upgrade. <span style="font-family:arial,sans-serif;font-size:13px">Currently we're using Lucene </span><span style="font-family:arial,sans-serif;font-size:13px">3</span><span style="font-family:arial,sans-serif;font-size:13px">.0.</span><span style="font-family:arial,sans-serif;font-size:13px">3, but index is build with 2.9</span><span style="font-family:arial,sans-serif;font-size:13px">. The 4.x lets you read older version index upto 3.0. If I can figure out difference between 2.9 vs 3.0 version, then we can decide if a hop to lucene 3.6 is necessary. I want to dig a little more into it, before a full blown discussion inside this version universe.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"> </span></div><div><span style="font-family:arial,sans-serif;font-size:13px">I am hoping to provide a transition phase (using config or two plugin option), so that existing index is not forced to upgrade. If we keep content of current field unchanged, it will reduce one variable.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Meanwhile if you guys can test the indexversion upgrade pullreq (</span><a href="https://github.com/crosswire/jsword/pull/79" target="_blank">https://github.com/crosswire/jsword/pull/79</a><span style="font-family:arial,sans-serif;font-size:13px">) & see if it meets the requirements of atleast your respective app's index upgrades, that will be great. </span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">In AndBible's usecase, it should help to prompt user to rebuild/download index of the one book (that is being search).</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">/sijo</span></div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Mon, May 5, 2014 at 3:17 PM, Chris Burrell <span dir="ltr"><<a href="mailto:christopher@burrell.me.uk" target="_blank">christopher@burrell.me.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi<div><br></div><div>Which version are we going for? </div><div><br></div><div>Not sure what you mean by 'best not to change the index structure'? Do you mean best to keep the current fields indexed with the current content? (if so, I agree, let's do this a step at a time).</div>
<div><br>Chris</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On 3 May 2014 21:32, Sijo Cherian <span dir="ltr"><<a href="mailto:sijo.cherian@gmail.com" target="_blank">sijo.cherian@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><p dir="ltr">FYI <br>
I just started working on upgrading code to newer lucene version, & keeping back compatibility.</p>
<p dir="ltr">More updates after some progress.<br>
It is best to not change index structure in default plugged-in version, for now.<span><font color="#888888"><br>
/sijo</font></span></p>
<br></div></div>_______________________________________________<br>
jsword-devel mailing list<br>
<a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br>
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
jsword-devel mailing list<br>
<a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br>
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Regards,<br>Sijo
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>Regards,<br>Sijo
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br>Regards,<br>Sijo
</font></span></div>
<br>_______________________________________________<br>
jsword-devel mailing list<br>
<a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a><br>
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
<br></blockquote></div><br></div>