[sword-devel] Issues with German Umlaut characters in SWORD trunk? [RESOLVED]

Sun Oct 4 15:40:23 EDT 2020

Tobias,

That's awesome to see that CMake is working out for building on Windows.
Two quick questions:

1) Are any of those CLI flags ones that we should include into the
CMakeLists.txt somewhere? I'm thinking specifically of the one where you
tell the system to export all symbols, but any others that should be
included or defaulted are also OK. Let me know and I can work on putting
them into the system to simplify your build process.

2) What version of CMake are you using? Someone on another thread is
struggling to build on Windows because of the default install directory
being set with \\ as the path separator and needed to switch it to /
instead. But they're on CMake 3.18.2, so I'm curious what version you're on
and if that is a new development in the CMake versions that they'll
understand Unix path separators on Windows.

--Greg

On Sun, Oct 4, 2020 at 12:15 PM Tobias Klein <contact at tklein.info> wrote:

> I managed to fix my SWORD build for Windows without changing anything in
> the sources. I've changed the way how the CMake options for ICU are set at
> the time when CMake is configured.
>
> Previously I didn't have the ICU_ROOT variable set. After changing that to
> the correct value, ICU is detected correctly. For those curious, this is
> the current build script I'm using for SWORD on Windows:
>
>
> https://github.com/tobias-klein/sword-build-win32/blob/71e23ea83ad93a066c2fc0264f4c633025234712/build_sword.bat
>
> After configuring the SWORD Windows build like this I'm not getting the
> "Umlaut" issue anymore that triggered this e-mail thread.
>
> Best regards,
> Tobias
> On 10/3/20 9:39 PM, Tobias Klein wrote:
>
> Ok, so it's not about the CMake version. Current build still has issues
> with the older CMake version.
>
> I've checked on changes in CMakeLists.txt.
>
> Greg changed this on July 28th (SVN Rev. 3773, Commit message "Simplify
> finding libraries").
>
> Diff:
> https://github.com/bibletime/crosswire-sword-mirror/commit/dfcb4e42fd61a295c6c0e60a62088c34fd139f76#diff-af3b638bc2a3e6c650974192a53c7291
>
> For MSVC the following ICU related configuration changed:
>
> Previously (before SVN Rev. 3773):
>
> FIND_PACKAGE(ICU REQUIRED)
> Now (from SVN Rev. 3773):
>
> FIND_PACKAGE(ICU COMPONENTS data i18n io uc)
>
> *This looks like the root-cause to my issue.*
>
> With the previous command ICU is correctly found resulting in
> Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
> (found version "65.1")
>
> With the new command ICU is only partially found and then subsequently
> ignored alltogether:
>
> -- SEARCHING FOR SYTEM PACKAGES
> 5089
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5089>--
> Found the following ICU libraries:
> 5090
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5090>--
> i18n (required)
> 5091
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5091>--
> uc (required)
> 5092
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5092>--
> The following ICU libraries were not found:
> 5093
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5093>--
> data (required)
> 5094
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5094>--
> io (required)
> 5095
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5095>--
> Failed to find all ICU components (missing: _ICU_REQUIRED_LIBS_FOUND)
> (found version "65.1")
>
> I guess I can now figure out why the data component and the io component
> are not found. In any case, before that change I could successfully build
> and link against ICU and in my application I did not get problems with
> German Umlaut's, which I now have ...
>
> I just double-checked the build with SVN Rev. 3772 (the version before
> Greg's change) and there ICU is still found correctly:
>
> -- SEARCHING FOR SYTEM PACKAGES
> 88
> <https://github.com/tobias-klein/sword-build-win32/runs/1203590932?check_suite_focus=true#step:12:88>--
> Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
> (found version "65.1")
>
> Any additional pointers? Are the ICU data and io components *actually*
> required by SWORD? I didn't seem to need them before.
>
> Best regards,
> Tobias
>
> On 10/3/20 8:09 PM, Tobias Klein wrote:
>
> When building SWORD based on SVN Rev. 3747 (May 18th) I get the following
> CMake output and with this SWORD version ICU *is still found correctly*.
>
> For both builds (ICU found vs. not found) I've been using the same ICU
> sources (Version 65, tag *release-65-1*,
> https://github.com/unicode-org/icu/tree/release-65-1).
>
> Since May 18th - has something changed in how the ICU library is
> found/detected?
>
> I will now check which CMake version is used on the GitHub build agents, I
> don't think I'm controlling that at the moment.
> -- SEARCHING FOR SYTEM PACKAGES
> 88
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:88>--
> Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
> (found version "65.1")
> 89
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:89>--
> Found CURL:
> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib (found
> version "7.68.0-DEV")
> 90
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:90>--
> System regex.h: No
> 91
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:91>--
>
> 92
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:92>--
> CONFIGURING SOURCE LIST
> 93
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:93>--
> ZLib: internal
> 94
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:94>--
> bzip2: system D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib
> 95
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:95>--
> xz: system D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib
> 96
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:96>--
> cURL: system
> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib and
> D:/a/sword-build-win32/sword-build-win32/curl/include
> 97
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:97>--
> CLucene: no
> 98
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:98>--
> PkgConfig: no
> 99
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:99>--
> ICU: yes
> D:\a\sword-build-win32\sword-build-win32\dist\lib\icudt.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuin.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuuc.lib
> and D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
> 100
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:100>--
> Regex.h: internal
> 101
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:101>--
> Building Shared library.
> 102
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:102>--
> Building Static library.
> 103
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:103>--
> Setting SOVERSION to 1.8.900
> 104
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:104>--
> ICU Version: 65.1
> 105
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:105>--
> Setting link libraries to
> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icudt.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuin.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuuc.lib
>
> On 10/3/20 7:29 PM, Tobias Klein wrote:
>
> Yes, I'm building SWORD with ICU. The build procedure has not changed
> recently.
>
> This is the batch script that builds SWORD on Windows:
>
>
> https://github.com/tobias-klein/sword-build-win32/blob/master/build_sword.bat
>
> However, I just found some interesting warnings in the build output from
> the most recent build:
> -- SEARCHING FOR SYTEM PACKAGES
> 5089
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5089>--
> Found the following ICU libraries:
> 5090
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5090>--
> i18n (required)
> 5091
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5091>--
> uc (required)
> 5092
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5092>--
> The following ICU libraries were not found:
> 5093
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5093>--
> data (required)
> 5094
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5094>--
> io (required)
> 5095
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5095>--
> Failed to find all ICU components (missing: _ICU_REQUIRED_LIBS_FOUND)
> (found version "65.1")
> 5096
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5096>--
> System regex.h: No
> 5097
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5097>--
>
> 5098
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5098>--
> CONFIGURING SOURCE LIST
> 5099
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5099>--
> ZLib: internal
> 5100
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5100>--
> bzip2: system D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib
> 5101
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5101>--
> xz: system D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib
> 5102
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5102>--
> cURL: system
> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib and
> D:/a/sword-build-win32/sword-build-win32/curl/include
> 5103
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5103>--
> CLucene: no
> 5104
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5104>--
> PkgConfig: no
> 5105
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5105>--
> ICU: no
> 5106
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5106>--
> Regex.h: internal
> 5107
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5107>--
> Building Shared library.
> 5108
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5108>--
> Building Static library.
> 5109
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5109>--
> Setting SOVERSION to 1.8.900
> 5110
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5110>--
> Setting link libraries to
> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib
>
> It says that certain ICU libraries were not found and then below under
> "CONFIGURING SOURCE LIST" it says ICU: no.
> I guess that could be the explanation. Unfortunately I don't have access
> to the build logs from May any longer ... maybe I can reproduce it anyway,
> I will keep looking.
>
> Best regards,
> Tobias
>
> On 10/3/20 1:44 PM, Troy A. Griffitts wrote:
>
> Hi Tobias, generally SWORD defaults to always return UTF-8. Depending on
> what you do with the string returned from your function, in Windows you may
> or may not get nice results. Also, to check, we do have multiple locales
> for German. The de-utf8 locale is preferred if SWORD is compiled with a
> StringMgr which understands Unicode. How are you compiling SWORD on
> Windows? With ICU?
>
> On October 3, 2020 10:55:32 AM GMT+02:00, Tobias Klein
> <contact at tklein.info> <contact at tklein.info> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> In node-sword-interface I have a method getSwordTranslation which does
>> the following:
>>
>>
>>
>> std::string translation = std::string(this->_localeMgr->translate(
>> originalString.c_str(), localeCode.c_str()));
>>
>> return translation;
>>
>>
>>
>> I have just found that this method does not do the right thing on Windows
>> any longer.
>>
>>
>>
>> When invoking my wrapper nsi.getSwordTranslation('Matthew', 'de') I now
>> get the following:
>>
>> Matth�us
>>
>>
>>
>> The expected result should be: Matthäus
>>
>>
>>
>> So the result is issues with German Umlaut characters. I only have this
>> issue on Windows, though – the Umlaut’s are returned correctly on Linux.
>>
>>
>>
>> The last SWORD revision I’ve been using without this issue is SVN Rev.
>> 3747 (from May 18th 2020).
>>
>>
>>
>> This would be a showstopper for me if it does turn out to be an issue in
>> SWORD.
>>
>>
>>
>> Best regards,
>> Tobias
>>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.orghttp://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.orghttp://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.orghttp://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20201004/e1a55524/attachment-0001.html>