[sword-devel] GlobalOptionFilter=UTF8GreekAccents

Thu Mar 20 22:53:01 EDT 2025

I think I’ve finished my impact analysis on SWORD search code. I ran out of time to do more. SWORD has several different search mechanisms. Some are affected by the proposed change. Some are not.

In all searches, the search request is normalized by the application by calling UTF8GreekAccents (and a few others) to remove accents and stripText is called to remove OSIS, GBF, ThML, TEI, …. markup and have simple markup for added words, overline and xref notes. If the search is case insensitive, it is converted to uppercase. The same is done for each piece of text that is pulled out of the module.

SEARCHTYPE_EXTERNAL - Lucene or Xapian
Both Lucene and Xapian, apply additional normalizations to their search requests in the same fashion as the text. I didn’t examine Xapian source so I’m not sure with Xapian. Lucene definitely does not have a problem. The results are the same with or without U+2019 as it is not a token character. I expect Xapian is similar.

SEARCHTYPE_ENTRYATTR - entryAttrib (eg. Word//Lemma./G1234/)	 (Lemma with dot means check components (Lemma.[1-9]) also)
I didn’t examine this. I’m not familiar with it. I’ll guess that it is no worse than the next three and might not have an issue.

SEARCHTYPE_PHRASE
The is a simple substring search.

SEARCHTYPE_REGEX
The search request is compiled into a regular expression (there are 3 variants depending on what regex libraries are used to build SWORD lib.)
There is special code to allow a RE to match across 2 verses.

SEARCHTYPE_MULTIWORD - multiword
The search request is split on spaces, ‘ ‘,  (not whitespace) into needles (as Troy calls them) and each needle is matched by simple substring in the haystack.
This is an AND search. All needles need to be found for there to be a hit.
There is special code to allow a request to match across 2 verses.

In these last 3 changing the UTF8GreekAccents to not remove U+2019 will change the user experience for search.
They will have to include it if it is in the text. This is no different than punctuation. which is not stripped from the text. This is no different from U+0027 which might be used in the text for the same purpose.
They will have to either copy/paste the character or figure out how to enter it. This is a non-obvious task. I outlined it in an earlier post.

When it comes to accented/unaccented texts, I thought I’d look for how unaccented texts handle the elision. Mark 2:17 is a good example of αλλα being shortened to αλλ followed by οι.
I looked at all the Greek (grc) modules at CrossWire that have Mark 2:17.

Here are the unaccented modules:
Antoniades Patriarchal Edition (1904/1912)
και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου αλλ οι κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν
The New Testament in the Original Greek: Byzantine Textform 2013
και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου αλλ οι κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν
Elzevir Textus Receptus (1624)
και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου αλλ οι κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν
Textus Receptus (1550/1894)
και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου αλλ οι κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν
Westcott and Hort with NA27/UBS4 variants
και ακουσας ο ιησους λεγει αυτοις [οτι] ου χρειαν εχουσιν οι ισχυοντες ιατρου αλλ οι κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους
Family 35
και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου αλλ οι κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν

Here are the accented modules:
Apostolic Bible Polyglot Greek Text
και ακούσας ο Ιησούς λέγει αυτοίς ου χρείαν έχουσιν οι ισχύοντες ιατρού αλλ΄ οι κακώς έχοντες ουκ ήλθον καλέσαι δικαίους αλλά αμαρτωλούς εις μετάνοιαν
Morphologically Parsed Greek New Testament based on the SBLGNT
καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς ⸀ὅτι Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ ἀλλ’ οἱ κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ⸀ἁμαρτωλούς.
Nestle GNT 1904
καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ ἀλλ’ οἱ κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ἁμαρτωλούς.
Nestle-Aland, Novum Testamentum Graece, 28th Revised Edition
καὶ ἀκούσας ὁ Ἰησοῦς λέγει °αὐτοῖς °¹[ὅτι] οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ ἀλλ ᾿ οἱ κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλ ᾿ ἁμαρτωλούς.
Nestle-Aland, Novum Testamentum Graece, 28th Revised Edition
καὶ ἀκούσας ὁ Ἰησοῦς λέγει °αὐτοῖς °¹[ὅτι] οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ ἀλλ ᾿ οἱ κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλ ᾿ ἁμαρτωλούς.
The Greek New Testament: SBL Edition
καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς ⸀ὅτι Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ ἀλλʼ οἱ κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ⸀ἁμαρτωλούς.
Statistical Restoration Greek New Testament
Καὶ ἀκούσας, ὁ ˚Ἰησοῦς λέγει αὐτοῖς, “Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ, ἀλλʼ οἱ κακῶς ἔχοντες. Οὐκ ἦλθον καλέσαι δικαίους, ἀλλὰ ἁμαρτωλούς.”
Tregelles' Greek New Testament
καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς, Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ, ἀλλ᾽ οἱ κακῶς ἔχοντες. οὐκ ἦλθον καλέσαι δικαίους, ἀλλὰ ἁμαρτωλούς.
Tischendorf's 8th edition GNT
	καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς· οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ ἀλλ’ οἱ κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ἁμαρτωλούς.

Note that all the unaccented Greek texts do not have αλλ΄ οι.

My recommendation is that we make no changes:
1) If we take the CrossWire unaccented Koine Greek modules as representative of the proper way to handle U+2019, then it should be stripped along with accents.
2) It changes current behavior that has not been an issue previously.
3) It is difficult for an end user to enter U+2019 and to know that it is a particular 1 of several glyphs that look the same.
4) Ancient texts, e.g. 4th-5th century, did not have accents or the elision mark.

In Him,
	DM

> On Mar 19, 2025, at 7:16 PM, DM Smith <dmsmith at crosswire.org> wrote:
> 
> David,
> 
> I’ve examined the use of Lucene in SWORD and it treats ’ U+2019 as a non-token character. SWORD uses the standard analyzer which is predicated on modern European languages with a nod to CJK. The ASCII apostrophe is looked at and in some circumstances it is a token character, but most often is ignored.
> 
> So from a Lucene perspective, the change does not affect it and would be safe.
> 
> While in the code, I found that Xapian is the preferred external search mechanism for indexed searches. From a quick look at Xapian and SWORD’s use of it, it is a far better search engine and has language awareness. I haven’t checked to see how it handles this elision in its tokenizer.
> 
> There are several other search mechanisms in SWORD that I’m checking out.
> 
> Aside, on my Mac, it is easy to input U+2019 from a USA keyboard input, but requires knowing how to do it and what it is. When I turned on Polytonic Greek as my keyboard input, it wasn’t possible. Apparently elision is rarely used in modern Greek. I had thought if anyone knew Koine (ancient) Greek and could readily type it, that they’d know how to input it. I’m not so sure.
> 
> Since you like your chat AI agents, ask it/them what the first and second level quote characters are for Greek. It surprised me. It’s not U+2019 or U+0027 for either.
> 
> DM
> 
> 
>> On Mar 18, 2025, at 12:55 PM, DM Smith <dmsmith at crosswire.org> wrote:
>> 
>> Apparently we are talking past each other.
>> 
>> I understand the argument. The elision mark is not an accent. It is a letter character when used between letters but a punctuation character (so a word break character) when at the beginning or end of words. It has been requested of the Unicode consortium to make it a letter character when it follows a Greek letter, but this has not been done. The goal of the request is that double clicking on the Greek word ending with U+2019 would also select the apostrophe in the same way that it does when it is in the middle of a character sequence.
>> 
>> I’m saying if the filter does double duty for both presentation and normalization of search requests, then both have to work well.
>> 
>> That U+2019 is visually similar to U+0027 and an end user will use the keyboard to type U+0027 when U+2019 is required. This will not work as the exact code point has to match.
>> 
>> Over the years, I’ve found that simple code changes often break other parts of the code.
>> 
>> I’ve requested to examine the search code in Xiphos to see if another filter is applied that strips U+2019. If so, then your request probably will work. However, if Xiphos needs to be changed in addition to SWORD lib to accommodate the change, then probably every frontend would need to be changed.
>> 
>> Also, I need to dig into the Lucene index creation and Lucene search to make sure it doesn’t require indexes to be rebuilt.
>> 
>> DM
>> 
>>> On Mar 18, 2025, at 3:21 AM, David Haslam <dfhdfh at protonmail.com> wrote:
>>> 
>>> We don't hide U+2019 in any other context.
>>> It's use in the KJV for possessives does not hinder search!
>>> 
>>> Why the obsession with search, when it has no bearing on my semantic argument?
>>> An elision mark is simply not an accent!!!  Why is this so hard to understand?
>>> 
>>> We wouldn't hide U+2019 in French if we saw it used frequently as an elision mark!!!
>>> Albeit most French modules to date simply use U+0027 (unlike our KJV).
>>> 
>>> Best regards,
>>> 
>>> David
>>> 
>>> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
>>> 
>>> On Tuesday, March 18th, 2025 at 1:40 AM, DM Smith <dmsmith at crosswire.org> wrote:
>>>> 
>>>> 
>>>>> On Mar 17, 2025, at 5:24 PM, David Haslam <dfhdfh at protonmail.com> wrote:
>>>>> 
>>>>> My argument is simple & straightforward.
>>>> 
>>>> Your argument is that display and search should have nothing to do with each other.
>>>> 
>>>>> 
>>>>> When you hide diacritics, you ought not to be hiding punctuation marks.
>>>>> 
>>>>> Why is this so contentious?
>>>> 
>>>> Software changes can have unintended consequences. It needs to be carefully considered.
>>>> 
>>>>> 
>>>>> In what world does how a module displays require that punctuation be hidden?
>>>> 
>>>> In a world where the filter is also used for search.
>>>>> 
>>>>> A quotation mark is not an accent! 
>>>>> 
>>>>> David
>>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Mar 17, 2025 at 21:06, DM Smith <dmsmith at crosswire.org <mailto:On Mon, Mar 17, 2025 at 21:06, DM Smith <<a href=>> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Mar 17, 2025, at 4:01 PM, Karl Kleinpaste <karl at kleinpaste.org> wrote:
>>>>>>> 
>>>>>>> That is, if search success depends on the specificity of whether accents or points are enabled, you're probably doing something wrong.
>>>>>> 
>>>>>> Absolutely.
>>>>>> 
>>>>>> What we are discussing is whether an apostrophe should be stripped out or not. The apostrophe on the keyboard is U+0027. The apostrophe in the Greek is recommended to be U+2019. They look nearly the same. There are a few other Unicode apostrophes that have the same appearance.
>>>>>> 
>>>>>> If the apostrophe isn’t stripped out, then the user input and the text have to agree on which of the several it is. There is no way for the end user to know which. I end up copying from the text to make sure I have the right one.
>>>>>> 
>>>>>> I believe the same filter that turns on and off the display of accents is used for searching. David is suggesting that the filter is taking out legitimate level 2 quotation marks from the display when it shouldn’t. I’m suggesting that it needs to remove them for the sake of the search.
>>>>>> 
>>>>>> DM
>>>>> _______________________________________________
>>>>> sword-devel mailing list: sword-devel at crosswire.org
>>>>> http://crosswire.org/mailman/listinfo/sword-devel
>>>>> Instructions to unsubscribe/change your settings at above page
>>>> 
>>> 
>>> _______________________________________________
>>> sword-devel mailing list: sword-devel at crosswire.org
>>> http://crosswire.org/mailman/listinfo/sword-devel
>>> Instructions to unsubscribe/change your settings at above page
>> 
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
> 
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20250320/2e2359b2/attachment-0001.htm>