<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">It doesn’t matter that a search doesn’t use Lucene. The principle is the same. The search request has to be normalized to the same form as the searched text. For example a case insensitive search normalizes both to a single case. If it isn’t done, even on the fly, then search will fail at times. As they say, “even a blind squirrel gets a nut sometimes."<div class=""><br class=""></div><div class="">Regarding Lucene there are mulitple different analyzers (that’s what does the normalization in Lucene). Each normalizes differently. Each has it’s own documentation. The analyzer that SWORD uses is suited and was developed for English texts. It is not appropriate for non-Latin texts. There is a multi-language analyzer that is much better, ICUAnalyzer, which follows UAX #29 for tokenization. For details see: <a href="https://issues.apache.org/jira/browse/LUCENE-1488" class="">https://issues.apache.org/jira/browse/LUCENE-1488</a> You’ll note that I participate in its development.</div><div class=""><br class=""></div><div class="">The osis2mod proclivity for NFC is for display.</div><div class=""><br class=""></div><div class="">DM<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 22, 2018, at 8:19 AM, David Haslam <<a href="mailto:dfhdfh@protonmail.com" class="">dfhdfh@protonmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Thanks DM,<br class=""></div><div class=""><br class=""></div><div class="">Not all searches make use of the Lucene index !<br class=""></div><div class=""><br class=""></div><div class="">e.g. In <b class="">Xiphos</b>, the advanced search panel gives the user a choice of which type of search.<br class=""></div><div class="">Lucene is only one of these mutually exclusive options.<br class=""></div><div class=""><br class=""></div><div class="">btw. Where is it documented that the creation of a Lucene search index normalizes the Unicode for the index?<br class=""></div><div class="">Do we know for certain that this would occur irrespective of whether normalization was suppressed during module build?<br class=""></div><div class="">i.e. With <b class="">osis2mod</b> option <b class=""> -N</b> do not convert UTF-8 or normalize UTF-8 to NFC<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="protonmail_signature_block"><div class="protonmail_signature_block-user"><div class="">Best regards,<br class=""></div><div class=""><br class=""></div><div class="">David<br class=""></div></div><div class=""><br class=""></div><div class="protonmail_signature_block-proton">Sent with <a href="https://protonmail.com/" target="_blank" class="">ProtonMail</a> Secure Email.<br class=""></div></div><div class=""><br class=""></div><div class="">‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐<br class=""></div><div class=""> On 22 March 2018 10:20 AM, DM Smith <<a href="mailto:dmsmith@crosswire.org" class="">dmsmith@crosswire.org</a>> wrote:<br class=""></div><div class=""> <br class=""></div><blockquote class="protonmail_quote" type="cite"><div class="">The requirement is not that the search is normalized to nfc but rather that it is normalized the same as the index. This should not be a front end issue.<br class=""></div><div class=""><br class=""></div><div class=""><div class="">Btw it doesn’t matter how Hebrew is stored in the module. Indexing should normalize it to a form that is internal to the engine. <br class=""></div><div class=""><br class=""></div><div class=""><div class="">— DM Smith<br class=""></div><div class="">From my phone. Brief. Weird autocorrections. <br class=""></div></div><div class=""><div class=""><br class=""></div><div class="">On Mar 22, 2018, at 5:22 AM, David Haslam <<a href="mailto:dfhdfh@protonmail.com" class="">dfhdfh@protonmail.com</a>> wrote:<br class=""></div></div><blockquote type="cite" class=""><div class=""><div class="">Dear all,<br class=""></div><div class=""><br class=""></div><div class="">Not all front-ends automatically <b class="">normalize the search string</b> to Unicode <b class="">NFC</b>.<br class=""></div><div class="">e.g.<br class=""></div><ul class=""><li class=""><b class="">Eloquent</b> does<br class=""></li><li class=""><b class="">Xiphos</b> does not<br class=""></li></ul><div class="">The data is incomplete for this feature in the table in our wiki page.<br class=""></div><div class=""><a href="https://wiki.crosswire.org/Choosing_a_SWORD_program#Search_and_Dictionary" class="">https://wiki.crosswire.org/Choosing_a_SWORD_program#Search_and_Dictionary</a><br class=""></div><div class=""><br class=""></div><div class=""><span class="size" style="font-size:16px">Please would other front-end app developers supply the missing information</span>. <i class="">Thanks</i>.<br class=""></div><div class=""><br class=""></div><div class=""><u class="">Further thought</u>:<br class=""></div><div class="">For front-ends that also have an <b class="">Advanced search</b> feature, would it not be a useful enhancement to have a <u class="">tick box option</u> for <b class="">Search string normalization</b>?<br class=""></div><div class="">Then if we do make any <u class="">Biblical Hebrew</u> modules with <i class=""><b class="">custom normalization</b></i>, search could at least still work for the "corner cases" in Hebrew, providing the user gave the proper input in the search box.<br class=""></div><div class=""><br class=""></div><div class=""><div class="">cf. The source text for the <b class="">WLC</b> at <a href="http://tanach.us/" class="">tanach.us</a> is <u class="">not</u> normalized to NFC, but our module is.<br class=""></div><div class=""><i class="">I'll refrain from going into a lot more detail here. There's an issue in our tracker that covers this.</i><br class=""></div></div><div class=""><br class=""></div><div class="protonmail_signature_block"><div class="protonmail_signature_block-user"><div class="">Best regards,<br class=""></div><div class=""><br class=""></div><div class="">David<br class=""></div></div><div class=""><br class=""></div><div class="protonmail_signature_block-proton">Sent with <a target="_blank" href="https://protonmail.com/" class="">ProtonMail</a> Secure Email.<br class=""></div></div><div class=""><br class=""></div></div></blockquote><blockquote type="cite" class=""><div class=""><div class=""><span class="">_______________________________________________</span><br class=""></div><div class=""><span class="">sword-devel mailing list: <a href="mailto:sword-devel@crosswire.org" class="">sword-devel@crosswire.org</a></span><br class=""></div><div class=""><span class=""><a href="http://www.crosswire.org/mailman/listinfo/sword-devel" class="">http://www.crosswire.org/mailman/listinfo/sword-devel</a></span><br class=""></div><div class=""><span class="">Instructions to unsubscribe/change your settings at above page</span><br class=""></div></div></blockquote></div></blockquote><div class=""><br class=""></div></div></blockquote></div><br class=""></div></body></html>