All,<br><br>I started working on improving search by normalizing accented characters(diacritic) in query.<br>Planning to use the lucene analysis contrib package for multi-lingual support(<a href="http://lucene.apache.org/java/docs/api/index.html">
http://lucene.apache.org/java/docs/api/index.html</a>).<br>I am Starting with a new AnalyzerFactory (and resource prop) that supplies appropriate instance of Analyzer based on bible language. Same analyzer <br>is used for indexing and for query parsing. Luckily our query always have a bible in context, so we have a language at query time.
<br><br>All corrections / suggestions are welocme and appreciated !<br><br>-- Tokenization Analysis--<br>-Current tokenization is based on SimpleAnalyzer(non-letter based tokenization). It breaks for Chinese/Korean bibles.
<br><br>--StopWord Analysis--<br>Question I asked myself: Is removing stop-word useful for us?<br><br>Common words skew the results or return overwhelming results.<br><br>Example Occurences in KJV of following words<br>the 24100
<br>of 18200<br>be 5500<br>you 2000<br>your 1300<br>unto 7360<br><br>Occurences in MKJV<br>you 8350<br>your 4600<br>shall 6400<br><br>Both during indexing & query parsing, the stop word are removed. So queries containing stopword terms will return smaller results
<br>and hits are influenced by other unique terms.<br><br><br>--Stemming Analysis--<br>Basics: <a href="http://www.comp.lancs.ac.uk/computing/research/stemming/general/">http://www.comp.lancs.ac.uk/computing/research/stemming/general/
</a><br><br>Is Stemming useful for us?<br><br>-It will be useful in many latin langs, to treat Accented & corresponding unaccented characters as same<br>- Along with stopword removal, it saves index space & search time.
<br>-KJV Examples where stemming can benefit:<br>Query:<br>+sin +lord +sight<br>Returns 15 results. But missed verses with 'sins'/'sinned' eg Deuteronomy 9:18 , 2 Kings 24:3<br><br>Query<br>+harp +sing<br>Returns 6 results. Missed harps/singers/singing as in 1 Kings 10:12, 1 Chr 13:8, 15:16 , 2Chr 5:12, 9:11, Nehemiah 12:27
<br><br>-If stemming is done by default, we can provide exact search operator, eg exact(singers). <br>In that case we can do a post-retrieval filter, or index both stemmed & unstemmed content(double space needed)<br><br>
-- Available functionality in Lucene Analyzer jar--<br><br>1. Both Stopwords & Stemming<br>Snowball based:<br>English(Porter or Lovins)<br>German<br>French<br>Dutch<br>Russian<br><br>Note: Another German stemming option (based on Joerg Caumanns paper):
org.apache.lucene.analysis.de.GermanAnalyzer<br>Manfred, you may be interested to look at the stopwords & stemming in it. I am guessing that snowball implementation <br>is preferable.<br><br>2. Stopwords only, no Stemming(from Lucene analysis contrib package)
<br>Czech<br>Greek (donot know if applicable to Modern or Ancient Greek Bible)<br><br>3. Only Stemming [Volunteers can contribute stopwords]<br>Snowball based:<br> Spanish <br> Portuguese <br> Italian <br> Swedish <br> Norwegian
<br> Danish <br> Finnish <br><br>4. Only tokenization (Lucene analysis contrib package)<br>ChineseAnalyzer (character tokenization): Better suited than CJKAnalyzer (which does 2 overlapping chinese character tokenization)
<br><br>-- Question I have now--<br><br>- Changing analyzers would involve current users to reindex all bible that they have indexed already. How do we manage that? Should we have a version framework for indexes, and BD can force a reindex when mismatch found.
<br><br>-Can we improve our search & index independent of sword?<br><br>-In long term, do we have any requirements & UI framework support to extend search to non-bible books?<br><br>Best,<br>Sijo<br><br>