<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On 18 April 2014 01:40, DM Smith <span dir="ltr"><<a href="mailto:dmsmith@crosswire.org" target="_blank">dmsmith@crosswire.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div class=""><div>
On Apr 17, 2014, at 12:09 PM, Chris Burrell <<a href="mailto:chris@burrell.me.uk" target="_blank">chris@burrell.me.uk</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">Hello<div><br></div><div>STEP uses stemming to improve search results, in some queries (whether on Sword modules or otherwise).</div>
</div></blockquote><div><br></div></div>Stemming is very useful. On occasion, there is a need for a non-stemmed search. Especially for theological purposes. But for general purpose searching it should be the default.</div>
<div><br></div></div></blockquote><div>Are you suggesting we have 'heading' being the stemmed search and fullHeading (or something like that) being the non-stemmed? I do think that by default however, we should have the normal search. We experimented with stemming in STEP by default and it can be quite confusing to look for a particular word and hit others. Stemming doesn't always work the way you expect.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">
<div></div><div>I've some times thought it'd be good to double index: stemmed and full word.</div><div><div class=""><br></div></div></div></blockquote><div>Double indexing is a need if you want both. The stem for genealogy resolves to genealogi (because of the plurals) which is why my search wasn't hit. We can't use the same field.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="">
<blockquote type="cite"><div dir="ltr"><div><br></div><div>There are currently 2 limitations in JSword, both of which could easily be fixed. Please let me know if you have concerns around me implementing both.</div>
<div><br></div><div>a- the frontend can't extend/control the use of indexes. I'm suggesting we add a registerFieldIndexer(fieldIndexer) with a simple interface: indexField(doc, osis). This would allow frontends to specify its own indexing. This would allow a frontend to index new things, or enable term vectors / store fields, etc. </div>
</div></blockquote><div><br></div></div>I'd really rather that we didn't go down this route. I don't mind plugin architecture as a way to experiment with different techniques, but I'd really rather that we all benefit from the changes.</div>
<div><br></div></div></blockquote><div>Fine.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word"><div></div><div><div class=""><blockquote type="cite"><div dir="ltr">
<div><br></div><div>b- Extend the LuceneIndex to have a stemmed version of the heading. We could replace the existing index, but that would mean all frontends will require re-indexing.</div></div></blockquote><div><br></div>
</div>I think the same manner that we index the main verse text should be applied to all text: intro, heading and verse text.</div><div><br></div></div></blockquote><div>Happy to do the change for all three.</div><div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div></div><div><div class="">
<blockquote type="cite"><div dir="ltr"><div><br></div><div>c- Had JSword been configured to 'STORE' the content of some fields, I would have used that for headings. For example, if the headings is stored in the index, STEP would not need to do an osis extract and XML transform to display to the user. It could come straight from the index. Two possibilities here: change the existing index field configuration, or duplicate into a different field.</div>
<div><br></div></div></blockquote><div><br></div></div>I think we should make store an option, possibly the standard.</div></div></blockquote><div>What I don't want to happen is end up in a situation where the Index is shared in different configurations by different apps. That would break the frontend. Even if you can ask, 'do you support', that's unnecessary complexity, that means that a user will have to re-index each book he has to support different front-ends. It also means that if a frontend forgets to ask whether some fields are indexed in a particular way, then he's going to have broken functionality in the frontend due to another frontend overriding the defaults. At this stage, I'd rather have app-specific indices. </div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">
<div><br></div><div>Right now the way we do the index prevents us from using Lucene to highlight the search hit. If that is STORE, then I'd be in favor of making STORE standard. I wonder if our stripping the text to no include OSIS before indexing will frustrate this change.</div>
<div><br></div></div></blockquote><div>Store is a requirement for highlighting (<a href="http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html">http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html</a> and <a href="http://wiki.apache.org/lucene-java/LuceneFAQ">http://wiki.apache.org/lucene-java/LuceneFAQ</a>).</div>
<div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">
<div></div><div>It still should be an option for the sake of devices that are disk limited.</div><div><div class=""><br><blockquote type="cite"><div dir="ltr">d- the other side of c- is that ideally multiple headings should be stored in multiple entries to the same field, rather than a concatenation of the field (doesn't much matter if it's only ANALYZED)</div>
</blockquote><div><br></div></div>Some verses have headings in the middle of the verse. Don't make the mistake of assuming an order of heading. Or that heading contains only pre-verse material or all pre-verse material.</div>
<div><div class=""><br></div></div></div></blockquote><div>I'm not making that mistake... All I'm saying is that headings should be stored in different entries in the same field.</div><div>doc.add(fieldName, heading1);</div>
<div><div>doc.add(fieldName, heading2);</div></div><div><div>doc.add(fieldName, heading3);</div></div><div><br></div><div>This means that you could retrieve one of the headings you want, rather than all. i.e. Psalm 3.1 Non-canon-heading Canon-heading could have 3 separate fields.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="">
<blockquote type="cite"><div dir="ltr">
<div><br></div><div><b>I only need one of a- or b- to be able to progress. Happy to do either. I don't need c- because I've worked around, but it would have been nice to have some control over that. </b></div><div>
<br></div><div>pros & cons:</div><div>a- more extensible in the future, other frontends don't benefit from enhancements</div><div>b- solves an immediate problem, but impacts all frontends (i.e. space used in index).</div>
<div><br></div><div>The only other bit in my mind is whether we need to ensure index-cross-application compatibility. I suspect some of this will tie in with the good work that Sijo has done on index management.</div></div>
</blockquote><div><br></div></div>The index management will be more critical with such a change. I've talked about having a manifest which defines the characteristics of the index. If we share an index created by two different systems, it will be important to "know" what an index supports.</div>
<div><br></div></div></blockquote><div>as described above, I'd like to avoid this. I don't think a frontend should have to worry about other frontends 'corrupting' the index (i.e. redefining fields, changing the store status, etc.). I'd rather my own index at that point. </div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">
<div></div><div>One of the changes that is being worked on is the update to a more recent version of Lucene. This affects how stemming is done. The way we are doing it now is deprecated and dropped.</div><div><div class="">
<br><blockquote type="cite"><div dir="ltr"><div>
<br></div><div>Let me know what your preferences are.</div></div></blockquote><div><br></div></div>Progress not perfection. Shared, configurable changes.</div><div><br><blockquote type="cite"><div dir="ltr"><div>Chris</div>
<div><br></div></div>
_______________________________________________<br>jsword-devel mailing list<br><a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
</blockquote></div><br></div></blockquote></div><br></div></div>