[sword-devel] Issues with German Umlaut characters in SWORD trunk? [RESOLVED]

Tobias Klein contact at tklein.info
Sun Oct 4 13:10:34 EDT 2020


I managed to fix my SWORD build for Windows without changing anything in 
the sources. I've changed the way how the CMake options for ICU are set 
at the time when CMake is configured.

Previously I didn't have the ICU_ROOT variable set. After changing that 
to the correct value, ICU is detected correctly. For those curious, this 
is the current build script I'm using for SWORD on Windows:

https://github.com/tobias-klein/sword-build-win32/blob/71e23ea83ad93a066c2fc0264f4c633025234712/build_sword.bat

After configuring the SWORD Windows build like this I'm not getting the 
"Umlaut" issue anymore that triggered this e-mail thread.

Best regards,
Tobias

On 10/3/20 9:39 PM, Tobias Klein wrote:
>
> Ok, so it's not about the CMake version. Current build still has 
> issues with the older CMake version.
>
> I've checked on changes in CMakeLists.txt.
>
> Greg changed this on July 28th (SVN Rev. 3773, Commit message 
> "Simplify finding libraries").
>
> Diff: 
> https://github.com/bibletime/crosswire-sword-mirror/commit/dfcb4e42fd61a295c6c0e60a62088c34fd139f76#diff-af3b638bc2a3e6c650974192a53c7291
>
> For MSVC the following ICU related configuration changed:
>
> Previously (before SVN Rev. 3773):
>
> FIND_PACKAGE(ICU REQUIRED)
>
> Now (from SVN Rev. 3773):
>
> FIND_PACKAGE(ICU COMPONENTSdata i18n io uc)
>
> _This looks like the root-cause to my issue._
>
> With the previous command ICU is correctly found resulting in
> Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include 
> (found version "65.1")
>
> With the new command ICU is only partially found and then subsequently 
> ignored alltogether:
>
> -- SEARCHING FOR SYTEM PACKAGES
> 5089 
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5089>-- 
> Found the following ICU libraries:
> 5090 
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5090>-- 
> i18n (required)
> 5091 
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5091>-- 
> uc (required)
> 5092 
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5092>-- 
> The following ICU libraries were not found:
> 5093 
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5093>-- 
> data (required)
> 5094 
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5094>-- 
> io (required)
> 5095 
> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5095>-- 
> Failed to find all ICU components (missing: _ICU_REQUIRED_LIBS_FOUND) 
> (found version "65.1")
>
> I guess I can now figure out why the data component and the io 
> component are not found. In any case, before that change I could 
> successfully build and link against ICU and in my application I did 
> not get problems with German Umlaut's, which I now have ...
>
> I just double-checked the build with SVN Rev. 3772 (the version before 
> Greg's change) and there ICU is still found correctly:
>
> -- SEARCHING FOR SYTEM PACKAGES
> 88 
> <https://github.com/tobias-klein/sword-build-win32/runs/1203590932?check_suite_focus=true#step:12:88>-- 
> Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include 
> (found version "65.1")
>
> Any additional pointers? Are the ICU data and io components _actually_ 
> required by SWORD? I didn't seem to need them before.
>
> Best regards,
> Tobias
>
> On 10/3/20 8:09 PM, Tobias Klein wrote:
>>
>> When building SWORD based on SVN Rev. 3747 (May 18th) I get the 
>> following CMake output and with this SWORD version ICU _is still 
>> found correctly_.
>>
>> For both builds (ICU found vs. not found) I've been using the same 
>> ICU sources (Version 65, tag /release-65-1/, 
>> https://github.com/unicode-org/icu/tree/release-65-1).
>>
>> Since May 18th - has something changed in how the ICU library is 
>> found/detected?
>>
>> I will now check which CMake version is used on the GitHub build 
>> agents, I don't think I'm controlling that at the moment.
>>
>> -- SEARCHING FOR SYTEM PACKAGES
>> 88 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:88>-- 
>> Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include 
>> (found version "65.1")
>> 89 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:89>-- 
>> Found CURL: 
>> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib 
>> (found version "7.68.0-DEV")
>> 90 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:90>-- 
>> System regex.h: No
>> 91 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:91>-- 
>>
>> 92 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:92>-- 
>> CONFIGURING SOURCE LIST
>> 93 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:93>-- 
>> ZLib: internal
>> 94 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:94>-- 
>> bzip2: system 
>> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib
>> 95 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:95>-- 
>> xz: system D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib
>> 96 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:96>-- 
>> cURL: system 
>> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib and 
>> D:/a/sword-build-win32/sword-build-win32/curl/include
>> 97 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:97>-- 
>> CLucene: no
>> 98 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:98>-- 
>> PkgConfig: no
>> 99 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:99>-- 
>> ICU: yes 
>> D:\a\sword-build-win32\sword-build-win32\dist\lib\icudt.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuin.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuuc.lib 
>> and D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
>> 100 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:100>-- 
>> Regex.h: internal
>> 101 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:101>-- 
>> Building Shared library.
>> 102 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:102>-- 
>> Building Static library.
>> 103 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:103>-- 
>> Setting SOVERSION to 1.8.900
>> 104 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:104>-- 
>> ICU Version: 65.1
>> 105 
>> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:105>-- 
>> Setting link libraries to 
>> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icudt.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuin.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuuc.lib
>>
>> On 10/3/20 7:29 PM, Tobias Klein wrote:
>>>
>>> Yes, I'm building SWORD with ICU. The build procedure has not 
>>> changed recently.
>>>
>>> This is the batch script that builds SWORD on Windows:
>>>
>>> https://github.com/tobias-klein/sword-build-win32/blob/master/build_sword.bat
>>>
>>> However, I just found some interesting warnings in the build output 
>>> from the most recent build:
>>>
>>> -- SEARCHING FOR SYTEM PACKAGES
>>> 5089 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5089>-- 
>>> Found the following ICU libraries:
>>> 5090 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5090>-- 
>>> i18n (required)
>>> 5091 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5091>-- 
>>> uc (required)
>>> 5092 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5092>-- 
>>> The following ICU libraries were not found:
>>> 5093 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5093>-- 
>>> data (required)
>>> 5094 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5094>-- 
>>> io (required)
>>> 5095 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5095>-- 
>>> Failed to find all ICU components (missing: 
>>> _ICU_REQUIRED_LIBS_FOUND) (found version "65.1")
>>> 5096 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5096>-- 
>>> System regex.h: No
>>> 5097 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5097>-- 
>>>
>>> 5098 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5098>-- 
>>> CONFIGURING SOURCE LIST
>>> 5099 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5099>-- 
>>> ZLib: internal
>>> 5100 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5100>-- 
>>> bzip2: system 
>>> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib
>>> 5101 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5101>-- 
>>> xz: system 
>>> D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib
>>> 5102 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5102>-- 
>>> cURL: system 
>>> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib 
>>> and D:/a/sword-build-win32/sword-build-win32/curl/include
>>> 5103 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5103>-- 
>>> CLucene: no
>>> 5104 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5104>-- 
>>> PkgConfig: no
>>> 5105 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5105>-- 
>>> ICU: no
>>> 5106 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5106>-- 
>>> Regex.h: internal
>>> 5107 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5107>-- 
>>> Building Shared library.
>>> 5108 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5108>-- 
>>> Building Static library.
>>> 5109 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5109>-- 
>>> Setting SOVERSION to 1.8.900
>>> 5110 
>>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5110>-- 
>>> Setting link libraries to 
>>> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib
>>>
>>> It says that certain ICU libraries were not found and then below 
>>> under "CONFIGURING SOURCE LIST" it says ICU: no.
>>> I guess that could be the explanation. Unfortunately I don't have 
>>> access to the build logs from May any longer ... maybe I can 
>>> reproduce it anyway, I will keep looking.
>>>
>>> Best regards,
>>> Tobias
>>>
>>> On 10/3/20 1:44 PM, Troy A. Griffitts wrote:
>>>> Hi Tobias, generally SWORD defaults to always return UTF-8. 
>>>> Depending on what you do with the string returned from your 
>>>> function, in Windows you may or may not get nice results. Also, to 
>>>> check, we do have multiple locales for German. The de-utf8 locale 
>>>> is preferred if SWORD is compiled with a StringMgr which 
>>>> understands Unicode. How are you compiling SWORD on Windows? With ICU?
>>>>
>>>> On October 3, 2020 10:55:32 AM GMT+02:00, Tobias Klein 
>>>> <contact at tklein.info> wrote:
>>>>
>>>>     Hi,
>>>>
>>>>     In node-sword-interface I have a method getSwordTranslation
>>>>     which does the following:
>>>>
>>>>     std::stringtranslation =
>>>>     std::string(this->_localeMgr->translate(originalString.c_str(),
>>>>     localeCode.c_str()));
>>>>
>>>>     returntranslation;
>>>>
>>>>     I have just found that this method does not do the right thing
>>>>     on Windows any longer.
>>>>
>>>>     When invoking my wrapper nsi.getSwordTranslation('Matthew',
>>>>     'de') I now get the following:
>>>>
>>>>     Matth�us
>>>>
>>>>     The expected result should be: Matthäus
>>>>
>>>>     So the result is issues with German Umlaut characters. I only
>>>>     have this issue on Windows, though – the Umlaut’s are returned
>>>>     correctly on Linux.
>>>>
>>>>     The last SWORD revision I’ve been using without this issue is
>>>>     SVN Rev. 3747 (from May 18th 2020).
>>>>
>>>>     This would be a showstopper for me if it does turn out to be an
>>>>     issue in SWORD.
>>>>
>>>>     Best regards,
>>>>     Tobias
>>>>
>>>>
>>>> -- 
>>>> Sent from my Android device with K-9 Mail. Please excuse my brevity. 
>>>
>>> _______________________________________________
>>> sword-devel mailing list:sword-devel at crosswire.org
>>> http://crosswire.org/mailman/listinfo/sword-devel
>>> Instructions to unsubscribe/change your settings at above page
>>
>> _______________________________________________
>> sword-devel mailing list:sword-devel at crosswire.org
>> http://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20201004/8b2aa6a3/attachment-0001.html>


More information about the sword-devel mailing list