[sword-devel] tests/testsuite/verseparsing-utf8

Mon Jul 27 01:49:13 EDT 2020

On Sun, Jul 26, 2020 at 10:34 AM Troy A. Griffitts <scribe at crosswire.org>
wrote:

> So Greg,
>
> I learned about cmake a bit today :)
>
> Nice work on the build system.
>
> I checked out a clean copy from source control, built with cmake,
> proceeded into the tests/testsuite folder
>
> ran: ./runall.sh
>
> and sure enough, it failed for me.
>
> Went back and re-read your sword/cmake/README to figure out how to build
> with debug.  Turned that on, rebuild, and stepped through the code.
>
> The difference:
>
> autotools does not include bindings/flatapi.cpp by default in the lib
>
> cmake does.
>
> The flatapi bindings injects its own StringMgr into SWORD to allow you to
> implement your own toUpperUTF C function (this is really the only necessary
> method SWORD requires to support Unicode at a basic level).
>
Aha! I figured there must be a discrepancy I couldn't spot. I did the first
pass of the cmake system before I knew very much about autotools at all.
I've gleaned quite a bit more about it, since then, but hadn't managed to
tease out that difference. Thanks!

> This was overriding the ICUStringMgr that was injected because of the
> build defines.
>
> I've enhanced this "feature" of flatapi.cpp to only inject its own
> StringMgr if StringMgr::hasUTF8Support == false.  Now flatapi.cpp can be
> included in libsword.so
>
> Anyway, in summary, if you update and build again and run the tests, they
> should work now.  Again, well done on the cmake system!
>
Confirmed - everything builds and runs perfectly with a freshly updated
copy of trunk.

As an interesting sidenote, all of the parsing code is about twice as fast
now in the testsuite. The regular verseparsing test was running a little
over 60 seconds before and it's now down around 35! The UTF8 one is down
from a few seconds to 1 second flat.

One thing about the shell scripts that I've learned - you might be best to
include a "set -e" in some of them. And, if they include pipes, a "set -o
pipefail" - which can be combined into "set -e -o pipefail". This will
ensure that the shell scripts will actually error out if a command isn't
found or exits improperly. Not necessary and might not be applicable to all
tests. But could be worth a consideration. I've had some tests give weird
results because they failed to write a file early on, but the shell script
continued going (due to having wonky directory permissions after
accidentally doing a build as root) and left odd behaviors in the test
result. Editing my local copy to include the "set -e" caused the whole
thing to stop at the moment of the write failure and made the test easier
to diagnose, instead of losing that output somewhere else.

--Greg

> Troy
>
>
> On 7/26/20 2:47 PM, Greg Hellings wrote:
>
>
>
> On Sun, Jul 26, 2020 at 7:29 AM Troy A. Griffitts <scribe at crosswire.org>
> wrote:
>
>> Greg, what's the output, or at least the couple lines of output from your
>> failed unit test?  It might help.  When I saw your note, I had suspected it
>> was because of something I had installed on my machine that was enabling
>> the test to pass (/etc/sword.conf) or something.  But I've removed what I
>> thought might be helping and I still pass all unit tests here on F32 using
>> autotools to build.
>>
> Oh yes, that's probably relevant!
>
> $ ./runtest.sh verseparsing-utf8
> Script failed at: (- bad output; + should have been)
> --- verseparsing-utf8.try 2020-07-26 08:45:12.077941691 -0400
> +++ verseparsing-utf8.good 2020-07-26 08:43:25.621662269 -0400
> @@ -1,7 +1,7 @@
> -Matthäus 2:3-12 de KJV ge 1:
> -Römer 2:13 de KJV ge 1:
> -Matthäus 1:2-Röm 3:13 de KJV ge 1:
> -1. Könige 2 de KJV ge 1:
> -1. Könige - 2. Könige; Markus 1:1; Matthäus 2:1; Matthäus; 1.Kön de KJV
> ge 1: Markus 1:1
> -1. Könige - 2. Könige; Markus 1:1; Matthäus 2:1; Matthäus; 1.Kön-2.Kön;I
> Kings-Matthäus de KJV ge 1: Markus 1:1; 1. K�nige 1:1-Markus 1:1
> -Maleachi 1:1 - Matthäus 2:1 de KJV ge 1: Maleachi 1:1-Offenbarung 22:21
> +Matthäus 2:3-12 de KJV ge 1: Matthäus 2:3-Matthäus 2:12
> +Römer 2:13 de KJV ge 1: Römer 2:13
> +Matthäus 1:2-Röm 3:13 de KJV ge 1: Matthäus 1:2-Römer 3:13
> +1. Könige 2 de KJV ge 1: 1. Könige 2:1-1. Könige 2:46
> +1. Könige - 2. Könige; Markus 1:1; Matthäus 2:1; Matthäus; 1.Kön de KJV
> ge 1: 1. Könige 1:1-2. Könige 25:30; Markus 1:1; Matthäus 2:1; Matthäus
> 1:1-Matthäus 28:20; 1. Könige 1:1-1. Könige 22:53
> +1. Könige - 2. Könige; Markus 1:1; Matthäus 2:1; Matthäus; 1.Kön-2.Kön;I
> Kings-Matthäus de KJV ge 1: 1. Könige 1:1-2. Könige 25:30; Markus 1:1;
> Matthäus 2:1; Matthäus 1:1-Matthäus 28:20; 1. Könige 1:1-2. Könige 25:30;
> 1. Könige 1:1-Matthäus 28:20
> +Maleachi 1:1 - Matthäus 2:1 de KJV ge 1: Maleachi 1:1-Matthäus 2:1
>
> It appears the test isn't even trying to parse the inputs? It's certainly
> not giving any output to make it look like it is.
>
> --Greg
>
>> Hope we can figure it out,
>>
>> Troy
>>
>>
>> On 7/25/20 5:46 AM, Greg Hellings wrote:
>>
>> Quick question - maybe a longer answer. Mostly for Troy, as you're
>> probably the only one familiar with this code:
>>
>> I'm trying to get the testsuit working in the CMake build. Things seem to
>> be going great. I can get 13/14 tests passing. But I've had a long-nagging
>> issue with getting verparsing-utf8 to pass in the CMake build. I'm building
>> with -D_ICU_ throughout, so it's not that the flag is missing. I'm also not
>> getting  passing results out of the autotools build, so this isn't just a
>> problem with CMake - although in the past I've had systems where I could
>> get the autotools build to work properly but the CMake one would not.
>>
>> I don't know enough about running a C++ debugger to dig into where the
>> problem lives. The verseparsing-utf8 is clear that the library needs ICU in
>> order for it to work, and it shows up in the list of linked libraries off
>> the sword library and in the compile flags.
>>
>> I assume the tests are still working for you? If so, any hints I can use
>> to track down why it's not working for me?
>>
>> --Greg
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.orghttp://www.crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.orghttp://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20200727/8fdb844b/attachment.html>