[sword-devel] GlobalOptionFilter=UTF8GreekAccents

Wed Mar 19 19:16:28 EDT 2025

David,

I’ve examined the use of Lucene in SWORD and it treats ’ U+2019 as a non-token character. SWORD uses the standard analyzer which is predicated on modern European languages with a nod to CJK. The ASCII apostrophe is looked at and in some circumstances it is a token character, but most often is ignored.

So from a Lucene perspective, the change does not affect it and would be safe.

While in the code, I found that Xapian is the preferred external search mechanism for indexed searches. From a quick look at Xapian and SWORD’s use of it, it is a far better search engine and has language awareness. I haven’t checked to see how it handles this elision in its tokenizer.

There are several other search mechanisms in SWORD that I’m checking out.

Aside, on my Mac, it is easy to input U+2019 from a USA keyboard input, but requires knowing how to do it and what it is. When I turned on Polytonic Greek as my keyboard input, it wasn’t possible. Apparently elision is rarely used in modern Greek. I had thought if anyone knew Koine (ancient) Greek and could readily type it, that they’d know how to input it. I’m not so sure.

Since you like your chat AI agents, ask it/them what the first and second level quote characters are for Greek. It surprised me. It’s not U+2019 or U+0027 for either.

DM

> On Mar 18, 2025, at 12:55 PM, DM Smith <dmsmith at crosswire.org> wrote:
> 
> Apparently we are talking past each other.
> 
> I understand the argument. The elision mark is not an accent. It is a letter character when used between letters but a punctuation character (so a word break character) when at the beginning or end of words. It has been requested of the Unicode consortium to make it a letter character when it follows a Greek letter, but this has not been done. The goal of the request is that double clicking on the Greek word ending with U+2019 would also select the apostrophe in the same way that it does when it is in the middle of a character sequence.
> 
> I’m saying if the filter does double duty for both presentation and normalization of search requests, then both have to work well.
> 
> That U+2019 is visually similar to U+0027 and an end user will use the keyboard to type U+0027 when U+2019 is required. This will not work as the exact code point has to match.
> 
> Over the years, I’ve found that simple code changes often break other parts of the code.
> 
> I’ve requested to examine the search code in Xiphos to see if another filter is applied that strips U+2019. If so, then your request probably will work. However, if Xiphos needs to be changed in addition to SWORD lib to accommodate the change, then probably every frontend would need to be changed.
> 
> Also, I need to dig into the Lucene index creation and Lucene search to make sure it doesn’t require indexes to be rebuilt.
> 
> DM
> 
>> On Mar 18, 2025, at 3:21 AM, David Haslam <dfhdfh at protonmail.com> wrote:
>> 
>> We don't hide U+2019 in any other context.
>> It's use in the KJV for possessives does not hinder search!
>> 
>> Why the obsession with search, when it has no bearing on my semantic argument?
>> An elision mark is simply not an accent!!!  Why is this so hard to understand?
>> 
>> We wouldn't hide U+2019 in French if we saw it used frequently as an elision mark!!!
>> Albeit most French modules to date simply use U+0027 (unlike our KJV).
>> 
>> Best regards,
>> 
>> David
>> 
>> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
>> 
>> On Tuesday, March 18th, 2025 at 1:40 AM, DM Smith <dmsmith at crosswire.org> wrote:
>>> 
>>> 
>>>> On Mar 17, 2025, at 5:24 PM, David Haslam <dfhdfh at protonmail.com> wrote:
>>>> 
>>>> My argument is simple & straightforward.
>>> 
>>> Your argument is that display and search should have nothing to do with each other.
>>> 
>>>> 
>>>> When you hide diacritics, you ought not to be hiding punctuation marks.
>>>> 
>>>> Why is this so contentious?
>>> 
>>> Software changes can have unintended consequences. It needs to be carefully considered.
>>> 
>>>> 
>>>> In what world does how a module displays require that punctuation be hidden?
>>> 
>>> In a world where the filter is also used for search.
>>>> 
>>>> A quotation mark is not an accent! 
>>>> 
>>>> David
>>>> 
>>>> 
>>>> 
>>>> On Mon, Mar 17, 2025 at 21:06, DM Smith <dmsmith at crosswire.org <mailto:On Mon, Mar 17, 2025 at 21:06, DM Smith <<a href=>> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mar 17, 2025, at 4:01 PM, Karl Kleinpaste <karl at kleinpaste.org> wrote:
>>>>>> 
>>>>>> That is, if search success depends on the specificity of whether accents or points are enabled, you're probably doing something wrong.
>>>>> 
>>>>> Absolutely.
>>>>> 
>>>>> What we are discussing is whether an apostrophe should be stripped out or not. The apostrophe on the keyboard is U+0027. The apostrophe in the Greek is recommended to be U+2019. They look nearly the same. There are a few other Unicode apostrophes that have the same appearance.
>>>>> 
>>>>> If the apostrophe isn’t stripped out, then the user input and the text have to agree on which of the several it is. There is no way for the end user to know which. I end up copying from the text to make sure I have the right one.
>>>>> 
>>>>> I believe the same filter that turns on and off the display of accents is used for searching. David is suggesting that the filter is taking out legitimate level 2 quotation marks from the display when it shouldn’t. I’m suggesting that it needs to remove them for the sake of the search.
>>>>> 
>>>>> DM
>>>> _______________________________________________
>>>> sword-devel mailing list: sword-devel at crosswire.org
>>>> http://crosswire.org/mailman/listinfo/sword-devel
>>>> Instructions to unsubscribe/change your settings at above page
>>> 
>> 
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
> 
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20250319/3cda70f9/attachment.htm>