[sword-devel] GlobalOptionFilter=UTF8GreekAccents and non-Greek modules

[DM Smith]

Hypothetical: What about mixed language texts such as a Greek/French lexicon?

DM

> On Feb 21, 2017, at 4:56 PM, Troy A. Griffitts <scribe at crosswire.org> wrote:
> 
> 
> Simply don't use the UTF-8 Greek Accent filter on non-Greek texts. As you have discovered there are accents used in Greek which are also used in other languages and adverse effects will be seen for these languages. The bottom line is simple. Only use the UTF-8 Greek Accents filter on UTF-8 Greek texts.
> 
> Hope this helps.
> 
> On February 21, 2017 2:45:24 PM MST, David Haslam <dfhmch at googlemail.com> wrote:
> These are the principal diacritics found in Biblical Greek that have to be
> removed with a UTF8GreekAccents filter.
> 
> The first five are general accents, not particular to Greek.
> It's on account of these that the filter should not be applied to non-Greek
> text.
> 
> U+0300 ̀ COMBINING GRAVE ACCENT
> U+0301 ́ COMBINING ACUTE ACCENT
> U+0308 ̈ COMBINING DIAERESIS
> U+0313 ̓ COMBINING COMMA ABOVE
> U+0314 ̔ COMBINING REVERSED COMMA ABOVE
> U+0342 ͂ COMBINING GREEK PERISPOMENI
> U+0343 ̓ COMBINING GREEK KORONIS
> U+0344 ̈́ COMBINING GREEK DIALYTIKA TONOS
> U+0345 ͅ COMBINING GREEK YPOGEGRAMMENI
> 
> No other diacritics or characters should be removed. 
> Though there are a few more combining accents in this block, they aren't
> really used in Biblical Greek.
> I am open to correction on this point.
> 
> e.g. The right single quotation mark (U+2019) is NOT a diacritic. It should
> not be removed.
> 
> Before any of these accents can be removed, they must first be separated
> from the Greek letters they are combined with. 
> 
> Although normalization to the decomposed form can produce this effect, as we
> have seen already, this can have undesirable side effects on any non-Greek
> text in the module that may happen to include combined or unusual
> characters.
> 
> It would therefore be more sensible to simply use a comprehensive mapping
> table that replaces each possible accented character by the corresponding
> letter in the Greek alphabet. In this way the filter can completely avoid
> the need to apply any Unicode normalization. 
> 
> The complete mapping table would have at least 130 rows. It will need to
> take into account that there are at least 75 possible combinations of a
> letter with two accents. There are none with three.
> 
> Any residual combining characters should also be removed, to cover the
> possibility that a module may have been intentionally made without
> normalizing the Greek source text by default to NFC.
> 
> That's my proposal. I can easily create such a mapping table that
> programmers can use.
> I can also readily test it with a bespoke TextPipe filter.
> 
> 
> Best regards,
> 
> David
> 
> 
> 
> 
> 
> --
> View this message in context: http://sword-dev.350566.n4.nabble.com/GlobalOptionFilter-UTF8GreekAccents-and-non-Greek-modules-tp4656719p4656765.html <http://sword-dev.350566.n4.nabble.com/GlobalOptionFilter-UTF8GreekAccents-and-non-Greek-modules-tp4656719p4656765.html>
> Sent from the SWORD Dev mailing list archive at Nabble.com <http://nabble.com/>.
> 
> 
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel <http://www.crosswire.org/mailman/listinfo/sword-devel>
> Instructions to unsubscribe/change your settings at above page
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20170221/9299cfd9/attachment-0001.html>