[sword-devel] DevTools:ICU & Normalization?

[Greg Hellings]

On Fri, Oct 28, 2011 at 10:28 AM, David Haslam <dfhmch at googlemail.com> wrote:
> FYI.  As a result of my posts in their forum arising from this topic,
> DataMystic have just released v8.9.8 of TextPipe.
>
> The release notes include:
>
> * Updated internal PCRE (Pattern Matching ) engine to v8.13 and support for
> Unicode 6.0.0.
> * Updated Unicode internal libraries to support Unicode 4.1 for
> Normalization etc.
>
> I have confirmed that TextPipe now Normalizes Burmese script to NFC with
> identical results to BabelPad.
> As an avid user of TextPipe Standard edition, for me this is nice step
> forward.
>
> Our *BurJudson* module was made with the source text normalized to an
> earlier version of Unicode.
>
> Unless one specifies otherwise (by means of the -N switch), osis2mod
> performs normalization to NFC.
>
> I would therefore recommend that precompiled SWORD utilities (especially
> those for Windows) should be built such that they adhere to the latest
> Unicode standard for Normalization.
>
> Likewise, front-end developers may have something to gain by pursuing this
> topic further, seeing as ICU has implications during module search, in
> regard to normalization of a search string, such that it ought to match how
> the module was normalized.

Front-end developers on Linux are largely limited by the distro they
reside on. SWORD supports, at the very least, ICU 4.0. Most modern
distros tend to include either ICU 4.6 or 4.8 - the latter being the
most up to date release available.

BibleTime does not use ICU at all, using Qt instead, so it is a rather
moot point with us on any operating system.  I do not know if Qt can
be compiled with an ICU backend or not, but it might be worth looking
into.

Xiphos on Windows, I believe, is distributed with ICU 4.0. This is
because that was the latest version of ICU that Matthew was able to
compile for Windows in the build environment Xiphos leverages.  I
believe Karl is also planning to use that same version. I have
successfully cross-compiled ICU 4.8 under Linux, but there are other
things hindering my ability to build Xiphos for Windows - most notably
there is very poor 64-bit Windows, cross-compile support in CLucene. I
have a patch which to fix that for CLucene 2, but I am waiting on Troy
to commit the CLucene 2 compatibility patch he and I developed for the
SWORD library, since I am unable to commit to that portion of the
SWORD repository. (Hint, hint, Troy. Waiting on you, still ;)

If he does that and I succeed at building with ICU 4.8 and CLucene 2,
then I will release a copy of the utilities with those libraries.  If
Chris updates his environment first, he will probably have more
success building natively with ICU 4.8 in VisualStudio.

--Greg