[sword-devel] Issues with German Umlaut characters in SWORD trunk?
Tobias Klein
contact at tklein.info
Sat Oct 3 15:39:25 EDT 2020
Ok, so it's not about the CMake version. Current build still has issues
with the older CMake version.
I've checked on changes in CMakeLists.txt.
Greg changed this on July 28th (SVN Rev. 3773, Commit message "Simplify
finding libraries").
Diff:
https://github.com/bibletime/crosswire-sword-mirror/commit/dfcb4e42fd61a295c6c0e60a62088c34fd139f76#diff-af3b638bc2a3e6c650974192a53c7291
For MSVC the following ICU related configuration changed:
Previously (before SVN Rev. 3773):
FIND_PACKAGE(ICU REQUIRED)
Now (from SVN Rev. 3773):
FIND_PACKAGE(ICU COMPONENTSdata i18n io uc)
_This looks like the root-cause to my issue._
With the previous command ICU is correctly found resulting in
Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
(found version "65.1")
With the new command ICU is only partially found and then subsequently
ignored alltogether:
-- SEARCHING FOR SYTEM PACKAGES
5089
<https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5089>--
Found the following ICU libraries:
5090
<https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5090>--
i18n (required)
5091
<https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5091>--
uc (required)
5092
<https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5092>--
The following ICU libraries were not found:
5093
<https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5093>--
data (required)
5094
<https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5094>--
io (required)
5095
<https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5095>--
Failed to find all ICU components (missing: _ICU_REQUIRED_LIBS_FOUND)
(found version "65.1")
I guess I can now figure out why the data component and the io component
are not found. In any case, before that change I could successfully
build and link against ICU and in my application I did not get problems
with German Umlaut's, which I now have ...
I just double-checked the build with SVN Rev. 3772 (the version before
Greg's change) and there ICU is still found correctly:
-- SEARCHING FOR SYTEM PACKAGES
88
<https://github.com/tobias-klein/sword-build-win32/runs/1203590932?check_suite_focus=true#step:12:88>--
Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
(found version "65.1")
Any additional pointers? Are the ICU data and io components _actually_
required by SWORD? I didn't seem to need them before.
Best regards,
Tobias
On 10/3/20 8:09 PM, Tobias Klein wrote:
>
> When building SWORD based on SVN Rev. 3747 (May 18th) I get the
> following CMake output and with this SWORD version ICU _is still found
> correctly_.
>
> For both builds (ICU found vs. not found) I've been using the same ICU
> sources (Version 65, tag /release-65-1/,
> https://github.com/unicode-org/icu/tree/release-65-1).
>
> Since May 18th - has something changed in how the ICU library is
> found/detected?
>
> I will now check which CMake version is used on the GitHub build
> agents, I don't think I'm controlling that at the moment.
>
> -- SEARCHING FOR SYTEM PACKAGES
> 88
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:88>--
> Found ICU: D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
> (found version "65.1")
> 89
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:89>--
> Found CURL:
> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib
> (found version "7.68.0-DEV")
> 90
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:90>--
> System regex.h: No
> 91
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:91>--
>
> 92
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:92>--
> CONFIGURING SOURCE LIST
> 93
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:93>--
> ZLib: internal
> 94
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:94>--
> bzip2: system
> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib
> 95
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:95>--
> xz: system D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib
> 96
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:96>--
> cURL: system
> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib and
> D:/a/sword-build-win32/sword-build-win32/curl/include
> 97
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:97>--
> CLucene: no
> 98
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:98>--
> PkgConfig: no
> 99
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:99>--
> ICU: yes
> D:\a\sword-build-win32\sword-build-win32\dist\lib\icudt.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuin.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuuc.lib
> and D:/a/sword-build-win32/sword-build-win32/icu/icu4c/include
> 100
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:100>--
> Regex.h: internal
> 101
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:101>--
> Building Shared library.
> 102
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:102>--
> Building Static library.
> 103
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:103>--
> Setting SOVERSION to 1.8.900
> 104
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:104>--
> ICU Version: 65.1
> 105
> <https://github.com/tobias-klein/sword-build-win32/runs/1203392535?check_suite_focus=true#step:12:105>--
> Setting link libraries to
> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icudt.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuin.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\icuuc.lib
>
> On 10/3/20 7:29 PM, Tobias Klein wrote:
>>
>> Yes, I'm building SWORD with ICU. The build procedure has not changed
>> recently.
>>
>> This is the batch script that builds SWORD on Windows:
>>
>> https://github.com/tobias-klein/sword-build-win32/blob/master/build_sword.bat
>>
>> However, I just found some interesting warnings in the build output
>> from the most recent build:
>>
>> -- SEARCHING FOR SYTEM PACKAGES
>> 5089
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5089>--
>> Found the following ICU libraries:
>> 5090
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5090>--
>> i18n (required)
>> 5091
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5091>--
>> uc (required)
>> 5092
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5092>--
>> The following ICU libraries were not found:
>> 5093
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5093>--
>> data (required)
>> 5094
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5094>--
>> io (required)
>> 5095
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5095>--
>> Failed to find all ICU components (missing: _ICU_REQUIRED_LIBS_FOUND)
>> (found version "65.1")
>> 5096
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5096>--
>> System regex.h: No
>> 5097
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5097>--
>>
>> 5098
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5098>--
>> CONFIGURING SOURCE LIST
>> 5099
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5099>--
>> ZLib: internal
>> 5100
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5100>--
>> bzip2: system
>> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib
>> 5101
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5101>--
>> xz: system D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib
>> 5102
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5102>--
>> cURL: system
>> D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib and
>> D:/a/sword-build-win32/sword-build-win32/curl/include
>> 5103
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5103>--
>> CLucene: no
>> 5104
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5104>--
>> PkgConfig: no
>> 5105
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5105>--
>> ICU: no
>> 5106
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5106>--
>> Regex.h: internal
>> 5107
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5107>--
>> Building Shared library.
>> 5108
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5108>--
>> Building Static library.
>> 5109
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5109>--
>> Setting SOVERSION to 1.8.900
>> 5110
>> <https://github.com/tobias-klein/sword-build-win32/runs/1145121405?check_suite_focus=true#step:6:5110>--
>> Setting link libraries to
>> D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib
>>
>> It says that certain ICU libraries were not found and then below
>> under "CONFIGURING SOURCE LIST" it says ICU: no.
>> I guess that could be the explanation. Unfortunately I don't have
>> access to the build logs from May any longer ... maybe I can
>> reproduce it anyway, I will keep looking.
>>
>> Best regards,
>> Tobias
>>
>> On 10/3/20 1:44 PM, Troy A. Griffitts wrote:
>>> Hi Tobias, generally SWORD defaults to always return UTF-8.
>>> Depending on what you do with the string returned from your
>>> function, in Windows you may or may not get nice results. Also, to
>>> check, we do have multiple locales for German. The de-utf8 locale is
>>> preferred if SWORD is compiled with a StringMgr which understands
>>> Unicode. How are you compiling SWORD on Windows? With ICU?
>>>
>>> On October 3, 2020 10:55:32 AM GMT+02:00, Tobias Klein
>>> <contact at tklein.info> wrote:
>>>
>>> Hi,
>>>
>>> In node-sword-interface I have a method getSwordTranslation
>>> which does the following:
>>>
>>> std::stringtranslation =
>>> std::string(this->_localeMgr->translate(originalString.c_str(),
>>> localeCode.c_str()));
>>>
>>> returntranslation;
>>>
>>> I have just found that this method does not do the right thing
>>> on Windows any longer.
>>>
>>> When invoking my wrapper nsi.getSwordTranslation('Matthew',
>>> 'de') I now get the following:
>>>
>>> Matth�us
>>>
>>> The expected result should be: Matthäus
>>>
>>> So the result is issues with German Umlaut characters. I only
>>> have this issue on Windows, though – the Umlaut’s are returned
>>> correctly on Linux.
>>>
>>> The last SWORD revision I’ve been using without this issue is
>>> SVN Rev. 3747 (from May 18th 2020).
>>>
>>> This would be a showstopper for me if it does turn out to be an
>>> issue in SWORD.
>>>
>>> Best regards,
>>> Tobias
>>>
>>>
>>> --
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>> _______________________________________________
>> sword-devel mailing list:sword-devel at crosswire.org
>> http://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20201003/7c439255/attachment-0001.html>
More information about the sword-devel
mailing list