[sword-devel] search failing in Hebrew modules

[Troy A. Griffitts]

Regarding languages with diacritics, accents, cantillation, etc...

The SWORD M.O. is to have one set of StripFilters that massage both:
o	the body of the text being searched
o	the target search string

so we can get sane results.

With Greek we've been fairly intentional to strip accents and ms markup 
from both module and search text input for our searching.  I would bet 
we still have some last minute code added somewhere which does special 
things if we're in a Greek text-- obviously this should be remedied.  I 
doubt we've done the same for Hebrew.  e.g., I would bet unaccented 
Greek searches would work fine in SWORDweb, but consonant-only Hebrew 
searches would not work.  In anycase, the proper way to make things work 
is to have appropriate StripFilter entries in the wlc.conf, and to be 
sure Xiphos is calling module.StripText(userInputSearchText) before 
calling SWORD's search mechanism to be sure we're comparing equivalent 
texts.

Does this make sense?

	-Troy.





Troy A. Griffitts wrote:
> SWORDWeb seems to work fine.  I'd appreciate it if we could have 
> construction fact input instead of useless statements like "it's SWORD's 
>  fault".  Thanks.
> 
> http://crosswire.org/study/wordsearchresults.jsp?searchTerm=שָׁמָיִם
> 
> Anyone willing to put the time into investigating if proper UTF-8 is 
> being sent into the SWORD engine from the copy and paste from Xiphos?
> 
>     -Troy.
> 
> 
> 
> Matthew Talbert wrote:
>>> I don't know for sure if this is the same bug, but I know that CLucene
>>> has severe issues (read: complete inability) with Unicode support.  If
>>> you are using a CLucene indexed module, this could definitely be a
>>> contributing factor to the problem.  In BibleTime we don't use SWORD's
>>> search features, we re-implement that ourselves with CLucene, and our
>>> result is a similar problem with Unicode modules that have indecies.
>>
>> The searches work nearly the same for indexed and non-indexed
>> searches, so it's SWORD, not clucene. I would be interested in hearing
>> what Unicode issues clucene has. The only one I recall is the
>> inability to prefix a search with a wildcard (which is very useful for
>> languages such as French).
>>
>> Matthew
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
> 
> 
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page