<div style="font-family: Arial, sans-serif; font-size: 14px;">Thanks DM,</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Very informative!</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important">Statistical Restoration Greek New Testament (<b>StatResGNT</b>) is an exception for a unique reason. Unlike any other Greek NT module, the project makes use of <b>quotation marks</b> for speech! </span><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important"><br></span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important">These were never in any original Greek MSS, so it's a form of reverse engineering.</span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important">cf. That's also the case with the added <b>Pilcrow</b> symbols <span>¶ </span>to mark paragraphs.</span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important"><br></span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important">When a user selects to <b>Hide</b> Greek Accents in the front-end (e.g. Xiphos), the level 2 quotations lose their closing marks in the displayed text. This is what led me to revisit the topic.</span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important"><br></span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span style="font-family:system-ui, sans-serif;display:inline !important"><u>Aside</u>: As a Windows user with <a href="https://www.babelstone.co.uk/Software/BabelPad.html" title="BabelPad">BabelPad</a> installed, I can enter any Unicode character into what I'm writing by looking it up in the <b>Character Map</b> tool, so I'd have no difficulty in inserting U+2019 into a search string. There are similar useful apps even for mobile platforms nowadays, such as <b>UnicodePad Pro</b> for iOS/iPadOS.</span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div>
<div class="protonmail_signature_block" style="font-family: Arial, sans-serif; font-size: 14px;">
<div class="protonmail_signature_block-user">
Best regards,<br><br>David
</div>
<div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div>
<div class="protonmail_signature_block-proton">
Sent with <a target="_blank" href="https://pr.tn/ref/SWXT9A5YZ67G">Proton Mail</a> secure email.
</div>
</div>
<div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div class="protonmail_quote">
On Friday, March 21st, 2025 at 2:53 AM, DM Smith <dmsmith@crosswire.org> wrote:<br>
<blockquote class="protonmail_quote" type="cite">
I think I’ve finished my impact analysis on SWORD search code. I ran out of time to do more. SWORD has several different search mechanisms. Some are affected by the proposed change. Some are not.<div><br></div><div>In all searches, the search request is normalized by the application by calling UTF8GreekAccents (and a few others) to remove accents and stripText is called to remove OSIS, GBF, ThML, TEI, …. markup and have simple markup for added words, overline and xref notes. If the search is case insensitive, it is converted to uppercase. The same is done for each piece of text that is pulled out of the module.</div><div><br></div><div><div>SEARCHTYPE_EXTERNAL - Lucene or Xapian</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>Both Lucene and Xapian, apply additional normalizations to their search requests in the same fashion as the text. I didn’t examine Xapian source so I’m not sure with Xapian. Lucene definitely does not have a problem. The results are the same with or without U+2019 as it is not a token character. I expect Xapian is similar.</div></div></blockquote><br><div><div>SEARCHTYPE_ENTRYATTR - entryAttrib (eg. Word//Lemma./G1234/)<span style="white-space: pre;" class="Apple-tab-span"> </span> (Lemma with dot means check components (Lemma.[1-9]) also)</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>I didn’t examine this. I’m not familiar with it. I’ll guess that it is no worse than the next three and might not have an issue.</div></div></blockquote><br><div><div>SEARCHTYPE_PHRASE</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">The is a simple substring search.</blockquote><div><br></div><div>SEARCHTYPE_REGEX</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div>The search request is compiled into a regular expression (there are 3 variants depending on what regex libraries are used to build SWORD lib.)</div><div>There is special code to allow a RE to match across 2 verses.</div><div><br></div></blockquote><div><div>SEARCHTYPE_MULTIWORD - multiword</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>The search request is split on spaces, ‘ ‘, (not whitespace) into needles (as Troy calls them) and each needle is matched by simple substring in the haystack.</div></div><div>This is an AND search. All needles need to be found for there to be a hit.</div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div>There is special code to allow a request to match across 2 verses.</div></div></div></blockquote><div><div><br></div><div>In these last 3 changing the UTF8GreekAccents to not remove U+2019 will change the user experience for search.</div><div><ul class="MailOutline"><li>They will have to include it if it is in the text. This is no different than punctuation. which is not stripped from the text. This is no different from U+0027 which might be used in the text for the same purpose.</li><li>They will have to either copy/paste the character or figure out how to enter it. This is a non-obvious task. I outlined it in an earlier post.</li></ul><div><br></div></div><div>When it comes to accented/unaccented texts, I thought I’d look for how unaccented texts handle the elision. Mark 2:17 is a good example of αλλα being shortened to αλλ followed by οι.</div><div>I looked at all the Greek (grc) modules at CrossWire that have Mark 2:17.</div><div><br></div><div>Here are the unaccented modules:</div><div>Antoniades Patriarchal Edition (1904/1912)</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div>και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου <b>αλλ οι</b> κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν</div></div></div></blockquote><div><div>The New Testament in the Original Greek: Byzantine Textform 2013</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου <b>αλλ οι</b> κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν</div></div></blockquote><div><div>Elzevir Textus Receptus (1624)</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου <b>αλλ οι</b> κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν</div></div></blockquote><div><div>Textus Receptus (1550/1894)</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου <b>αλλ οι</b> κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν</div></div></blockquote><div><div>Westcott and Hort with NA27/UBS4 variants</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>και ακουσας ο ιησους λεγει αυτοις [οτι] ου χρειαν εχουσιν οι ισχυοντες ιατρου <b>αλλ οι </b>κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους</div></div></blockquote><div><div>Family 35</div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>και ακουσας ο ιησους λεγει αυτοις ου χρειαν εχουσιν οι ισχυοντες ιατρου <b>αλλ οι</b> κακως εχοντες ουκ ηλθον καλεσαι δικαιους αλλα αμαρτωλους εις μετανοιαν</div></div></blockquote><div><div><div><div><br></div><div>Here are the accented modules:</div><div>Apostolic Bible Polyglot Greek Text</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">και ακούσας ο Ιησούς λέγει αυτοίς ου χρείαν έχουσιν οι ισχύοντες ιατρού <b>αλλ΄ οι</b> κακώς έχοντες ουκ ήλθον καλέσαι δικαίους αλλά αμαρτωλούς εις μετάνοιαν</blockquote><div></div><div>Morphologically Parsed Greek New Testament based on the SBLGNT</div></div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς ⸀ὅτι Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ <b>ἀλλ’ οἱ</b> κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ⸀ἁμαρτωλούς.</blockquote><div>Nestle GNT 1904</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ <b>ἀλλ’ οἱ</b> κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ἁμαρτωλούς.</blockquote><div>Nestle-Aland, Novum Testamentum Graece, 28th Revised Edition</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">καὶ ἀκούσας ὁ Ἰησοῦς λέγει °αὐτοῖς °¹[ὅτι] οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ <b>ἀλλ ᾿ οἱ</b> κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλ ᾿ ἁμαρτωλούς.</blockquote><div>Nestle-Aland, Novum Testamentum Graece, 28th Revised Edition</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">καὶ ἀκούσας ὁ Ἰησοῦς λέγει °αὐτοῖς °¹[ὅτι] οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ <b>ἀλλ ᾿ οἱ</b> κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλ ᾿ ἁμαρτωλούς.</blockquote><div>The Greek New Testament: SBL Edition</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς ⸀ὅτι Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ <b>ἀλλʼ οἱ</b> κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ⸀ἁμαρτωλούς.</blockquote><div>Statistical Restoration Greek New Testament</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">Καὶ ἀκούσας, ὁ ˚Ἰησοῦς λέγει αὐτοῖς, “Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ, <b>ἀλλʼ οἱ</b> κακῶς ἔχοντες. Οὐκ ἦλθον καλέσαι δικαίους, ἀλλὰ ἁμαρτωλούς.”</blockquote><div>Tregelles' Greek New Testament</div><blockquote style="margin: 0px 0px 0px 40px; border: medium; padding: 0px;">καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς, Οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ, <b>ἀλλ᾽ οἱ</b> κακῶς ἔχοντες. οὐκ ἦλθον καλέσαι δικαίους, ἀλλὰ ἁμαρτωλούς.</blockquote><div></div></div><div><div><div>Tischendorf's 8th edition GNT</div><div><span style="white-space: pre;" class="Apple-tab-span"> </span>καὶ ἀκούσας ὁ Ἰησοῦς λέγει αὐτοῖς· οὐ χρείαν ἔχουσιν οἱ ἰσχύοντες ἰατροῦ <b>ἀλλ’ οἱ</b> κακῶς ἔχοντες· οὐκ ἦλθον καλέσαι δικαίους ἀλλὰ ἁμαρτωλούς.</div><div><br></div><div>Note that all the unaccented Greek texts do not have <b>αλλ΄ </b><b>οι</b>.</div><div><br></div><div>My recommendation is that we make no changes:</div><div>1) If we take the CrossWire unaccented Koine Greek modules as representative of the proper way to handle U+2019, then it should be stripped along with accents.</div><div>2) It changes current behavior that has not been an issue previously.</div><div>3) It is difficult for an end user to enter U+2019 and to know that it is a particular 1 of several glyphs that look the same.</div><div>4) Ancient texts, e.g. 4th-5th century, did not have accents or the elision mark.</div><div><br></div><div>In Him,</div><div><span style="white-space:pre" class="Apple-tab-span"> </span>DM</div><div><br></div><blockquote type="cite"><div>On Mar 19, 2025, at 7:16 PM, DM Smith <dmsmith@crosswire.org> wrote:</div><br class="Apple-interchange-newline"><div><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">David,<div><br></div><div>I’ve examined the use of Lucene in SWORD and it treats ’ U+2019 as a non-token character. SWORD uses the standard analyzer which is predicated on modern European languages with a nod to CJK. The ASCII apostrophe is looked at and in some circumstances it is a token character, but most often is ignored.<div><br></div><div>So from a Lucene perspective, the change does not affect it and would be safe.</div><div><br></div><div>While in the code, I found that Xapian is the preferred external search mechanism for indexed searches. From a quick look at Xapian and SWORD’s use of it, it is a far better search engine and has language awareness. I haven’t checked to see how it handles this elision in its tokenizer.</div><div><br></div><div>There are several other search mechanisms in SWORD that I’m checking out.</div><div><br></div><div>Aside, on my Mac, it is easy to input U+2019 from a USA keyboard input, but requires knowing how to do it and what it is. When I turned on Polytonic Greek as my keyboard input, it wasn’t possible. Apparently elision is rarely used in modern Greek. I had thought if anyone knew Koine (ancient) Greek and could readily type it, that they’d know how to input it. I’m not so sure.</div><div><br></div><div>Since you like your chat AI agents, ask it/them what the first and second level quote characters are for Greek. It surprised me. It’s not U+2019 or U+0027 for either.</div><div><br></div><div>DM</div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Mar 18, 2025, at 12:55 PM, DM Smith <dmsmith@crosswire.org> wrote:</div><br class="Apple-interchange-newline"><div><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Apparently we are talking past each other.<div><br></div><div>I understand the argument. The elision mark is not an accent. It is a letter character when used between letters but a punctuation character (so a word break character) when at the beginning or end of words. It has been requested of the Unicode consortium to make it a letter character when it follows a Greek letter, but this has not been done. The goal of the request is that double clicking on the Greek word ending with U+2019 would also select the apostrophe in the same way that it does when it is in the middle of a character sequence.</div><div><br></div><div>I’m saying if the filter does double duty for both presentation and normalization of search requests, then both have to work well.</div><div><br></div><div>That U+2019 is visually similar to U+0027 and an end user will use the keyboard to type U+0027 when U+2019 is required. This will not work as the exact code point has to match.</div><div><br></div><div>Over the years, I’ve found that simple code changes often break other parts of the code.</div><div><br></div><div>I’ve requested to examine the search code in Xiphos to see if another filter is applied that strips U+2019. If so, then your request probably will work. However, if Xiphos needs to be changed in addition to SWORD lib to accommodate the change, then probably every frontend would need to be changed.</div><div><br></div><div>Also, I need to dig into the Lucene index creation and Lucene search to make sure it doesn’t require indexes to be rebuilt.</div><div><br></div><div>DM<br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Mar 18, 2025, at 3:21 AM, David Haslam <dfhdfh@protonmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div style="font-family: Arial, sans-serif; font-size: 14px;">We don't hide U+2019 in any other context.</div><div style="font-family: Arial, sans-serif; font-size: 14px;">It's use in the KJV for possessives does not hinder search!</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Why the obsession with search, when it has no bearing on my semantic argument?</div><div style="font-family: Arial, sans-serif; font-size: 14px;">An elision mark is simply not an accent!!! Why is this so hard to understand?</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">We wouldn't hide U+2019 in French if we saw it used frequently as an elision mark!!!</div><div style="font-family: Arial, sans-serif; font-size: 14px;">Albeit most French modules to date simply use U+0027 (unlike our KJV).</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div>
<div style="font-family: Arial, sans-serif; font-size: 14px;" class="protonmail_signature_block">
<div class="protonmail_signature_block-user">
Best regards,<br><br>David
</div>
<div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div>
<div class="protonmail_signature_block-proton">
Sent with <a href="https://pr.tn/ref/SWXT9A5YZ67G" target="_blank" rel="noreferrer nofollow noopener">Proton Mail</a> secure email.
</div>
</div>
<div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div class="protonmail_quote">
On Tuesday, March 18th, 2025 at 1:40 AM, DM Smith <dmsmith@crosswire.org> wrote:<br>
<blockquote type="cite" class="protonmail_quote">
<br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Mar 17, 2025, at 5:24 PM, David Haslam <dfhdfh@protonmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div> <div dir="auto">My argument is simple & straightforward.</div></div></div></blockquote><div><br></div>Your argument is that display and search should have nothing to do with each other.</div><div><br><blockquote type="cite"><div><div><div dir="auto"><br></div><div dir="auto">When you hide diacritics, you ought not to be hiding punctuation marks.</div><div dir="auto"><br></div><div dir="auto">Why is this so contentious?</div></div></div></blockquote><div><br></div>Software changes can have unintended consequences. It needs to be carefully considered.</div><div><br><blockquote type="cite"><div><div><div dir="auto"><br></div><div dir="auto">In what world does how a module displays require that punctuation be hidden?</div></div></div></blockquote><div><br></div>In a world where the filter is also used for search.<br><blockquote type="cite"><div><div><div dir="auto"><br></div><div dir="auto">A quotation mark is not an accent! </div><div dir="auto"><br></div><div dir="auto">David</div><div><br></div> <div><br></div><div><br></div>On Mon, Mar 17, 2025 at 21:06, DM Smith <<a rel="noreferrer nofollow noopener" href="mailto:On Mon, Mar 17, 2025 at 21:06, DM Smith <<a href=">dmsmith@crosswire.org</a>> wrote:<blockquote type="cite" class="protonmail_quote"> <br id="lineBreakAtBeginningOfMessage">
<div>
<br>
<blockquote type="cite">
<div>
On Mar 17, 2025, at 4:01 PM, Karl Kleinpaste <karl@kleinpaste.org> wrote:
</div>
<br class="Apple-interchange-newline">
<div>
<span style="caret-color: rgb(0, 0, 0); font-family: FreeSerif; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">That is, if search success depends on the specificity of whether accents or points are enabled, you're probably doing something wrong.</span>
<br style="caret-color: rgb(0, 0, 0); font-family: FreeSerif; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
</div>
</blockquote>
<br>
</div>
<div>
Absolutely.
</div>
<div>
<br>
</div>
<div>
What we are discussing is whether an apostrophe should be stripped out or not. The apostrophe on the keyboard is U+0027. The apostrophe in the Greek is recommended to be U+2019. They look nearly the same. There are a few other Unicode apostrophes that have the same appearance.
</div>
<div>
<br>
</div>
<div>
If the apostrophe isn’t stripped out, then the user input and the text have to agree on which of the several it is. There is no way for the end user to know which. I end up copying from the text to make sure I have the right one.
</div>
<div>
<br>
</div>
<div>
I believe the same filter that turns on and off the display of accents is used for searching. David is suggesting that the filter is taking out legitimate level 2 quotation marks from the display when it shouldn’t. I’m suggesting that it needs to remove them for the sake of the search.
</div>
<div>
<br>
</div>
<div>
DM
</div></blockquote></div>_______________________________________________<br>sword-devel mailing list: sword-devel@crosswire.org<br>http://crosswire.org/mailman/listinfo/sword-devel<br>Instructions to unsubscribe/change your settings at above page<br></div></blockquote></div><br>
</blockquote><br>
</div>_______________________________________________<br>sword-devel mailing list: sword-devel@crosswire.org<br>http://crosswire.org/mailman/listinfo/sword-devel<br>Instructions to unsubscribe/change your settings at above page<br></div></blockquote></div><br></div></div></div>_______________________________________________<br>sword-devel mailing list: sword-devel@crosswire.org<br>http://crosswire.org/mailman/listinfo/sword-devel<br>Instructions to unsubscribe/change your settings at above page<br></div></blockquote></div><br></div></div></div></div>_______________________________________________<br>sword-devel mailing list: sword-devel@crosswire.org<br>http://crosswire.org/mailman/listinfo/sword-devel<br>Instructions to unsubscribe/change your settings at above page<br></div></blockquote></div><br></div></div>
</blockquote><br>
</div>