<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear Tobias,</p>
<p>Thank you for all the great information. This enabled me to
isolate the change which caused the issue.</p>
<p>So, for a bit of background, SWORD has no calls to mark critical
sections which might be problematic for re-entrant usage. This
has been due to the many implementations of threading across many
different platforms over the years, before C++11. But, as a
policy to support clients which desire to use SWORD in a
multithreaded manner, we do our best to make this safe by advising
clients to use separate SWMgr instances per thread. There are
still some shared objects in this scenario, but we do our best to
do all the writing to these shared objects upon initialization.
We broke this rule in commit 2760, which is what caused your
problem. SWORD have a facility to pool open file handles, to help
OSs which have small open file handle limits. This work is done
in FileMgr. Recently, to support Windows Unicode path names (the
commit you found which breaks your multithreaded use), we rounded
up all remaining native file IO calls and replaced them to used
FileMgr for the IO and then extended FileMgr to handle Windows
Unicode paths in a Windows-specific manner. One of these changes
was in CURLFTPTransport, which is where you are having the issue.
The problem is that, where previously this class was directly
opening a FILE to do its writing, commit 2760 changed this to use
FileMgr to open the file, which involved the SWORD-wide file
handle pool, and since we are create a new file, we are always
writing to this shared pool container, which is not threadsafe.
My guess is that you have two threads trying to update the pool
container at exactly the same time. Using the file handle pool is
usually safe, because SWMgr "opens" all of its file handles on
initialization (these are not actually opening OS file handles,
but instead updating the file handle pool container with proxy
objects which delay actual OS open to on-demand, but the point is
this instance of shared file handle pool container writing is done
on creation of the SWMgr, afterward, the shared resource file
handle pool is only read and each object in the pool is owned by
only 1 thread if the "each thread must have its own SWMgr" rule is
followed.<br>
</p>
<p>Regardless of the details. I believe I have committed a fix for
you. In short, I have changed CURLFTPTransport to follow our rule
to avoid writing to shared objects when we might be re-entrant.
Here we now use FileMgr's methods which isolate OS implementation,
but not FileMgr's file handle pool (as it did not previously use
the pool before this commit). This should allow this to still
take advantage of the Windows OS-specific implementation, and also
avoid the critical section. Can you please try SVN head and let
me know if we are back to 20 out of 20 successes?</p>
<p>Thanks again for the very helpful debug log and exact revision
where failure began.</p>
<p>Troy<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 10/13/20 10:08 PM, Tobias Klein
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:3258796d-a77a-0009-b0b3-821cda1c89a4@tklein.info">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>I managed to get a backtrace to a segmentation fault using GDB.</p>
<p>It seems like the crash is happening in sword::FileMgr::open(
...</p>
<p>The starting point is sword::InstallMgr::refreshRemoteSource as
I was writing before.</p>
<p>Best regards,<br>
Tobias<br>
</p>
<p>Program received signal SIGSEGV, Segmentation fault.<br>
[Switching to Thread 0x7f1af3fff700 (LWP 220833)]<br>
0x00007f1b027045a4 in sword::FileMgr::open(char const*, int,
int, bool) () from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
(gdb) backtrace<br>
#0 0x00007f1b027045a4 in sword::FileMgr::open(char const*, int,
int, bool) () from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#1 0x00007f1b0276ad7b in sword::(anonymous
namespace)::my_fwrite(void*, unsigned long, unsigned long,
void*) ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#2 0x00007f1b180626bf in ?? () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#3 0x00007f1b18074a2b in ?? () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#4 0x00007f1b1807e2e4 in ?? () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#5 0x00007f1b1807f6f9 in curl_multi_perform () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#6 0x00007f1b18075d13 in curl_easy_perform () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#7 0x00007f1b0276b683 in sword::CURLFTPTransport::getURL(char
const*, char const*, sword::SWBuf*) () from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#8 0x00007f1b0271d5d2 in
sword::InstallMgr::remoteCopy(sword::InstallSource*, char
const*, char const*, bool, char const*) ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#9 0x00007f1b0271edc7 in
sword::InstallMgr::refreshRemoteSource(sword::InstallSource*) ()
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#10 0x00007f1b026ad734 in
RepositoryInterface::refreshIndividualRemoteSource(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::function<void (unsigned int)>*) ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#11 0x00007f1b026b17dd in
std::thread::_State_impl<std::thread::_Invoker<std::tuple<int
(RepositoryInterface::*)(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::function<void (unsigned int)>*),
RepositoryInterface*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::function<void (unsigned int)>*> >
>::_M_run() ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#12 0x00007f1b1d622cb4 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6<br>
#13 0x00007f1b1e20a609 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0<br>
#14 0x00007f1b1e131103 in clone () from
/lib/x86_64-linux-gnu/libc.so.6<br>
</p>
<div class="moz-cite-prefix">On 10/13/20 1:07 PM, Tobias Klein
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:03bca06c-8c98-434a-ff32-3947e285961a@tklein.info">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>Hi Troy,</p>
<p>I tested more SVN revisions of SWORD trunk (starting from my
stable version until I hit the bug) and I can now say that</p>
<p>SVN Rev. 3759 is the last SVN revision that works without
hanging for the below mentioned scenario. (20 out of 20 tests
successful)<br>
</p>
<p>SVN Rev. 3760 is the first SVN revision where the hanging
occurs. The commit message is "First cut at better isolation
of FileIO to FileMgr and providing a WIN32 impl with works
with wchar_t".</p>
<p>Modified files:<br>
include/filemgr.h<br>
include/swbuf.h<br>
lib/bcppmake/libsword.bpr<br>
src/mgr/curlftpt.cpp<br>
src/mgr/curlhttpt.cpp<br>
src/mgr/filemgr.cpp<br>
src/mgr/installmgr.cpp<br>
src/mgr/swmgr.cpp<br>
src/utilfuns/utilstr.cpp<br>
<br>
Maybe this helps to find the root-cause.<br>
<br>
Best regards,<br>
Tobias<br>
</p>
<div class="moz-cite-prefix">On 10/12/20 9:20 PM, Tobias Klein
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:fcc4672a-feaf-4eec-9f7c-a0acdc57faf0@tklein.info">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>I'll see whether I can collect a stack trace. It may take
some time until I have it.</p>
<p>The multi-threaded "remote source refreshing" worked
without issues until recently.<br>
</p>
<p>Here is the code of the function that does the actual work
in a thread.<br>
See <a class="moz-txt-link-freetext"
href="https://github.com/tobias-klein/node-sword-interface/blob/787160ccb4b3bab2a762d22f74031c7237edc803/src/sword_backend/repository_interface.cpp#L105"
moz-do-not-send="true">https://github.com/tobias-klein/node-sword-interface/blob/787160ccb4b3bab2a762d22f74031c7237edc803/src/sword_backend/repository_interface.cpp#L105</a>.</p>
<div style="color: #d4d4d4;background-color: #1e1e1e;font-family: 'Droid Sans Mono', 'monospace', monospace, 'Droid Sans Fallback';font-weight: normal;font-size: 14px;line-height: 19px;white-space: pre;"><div><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">RepositoryInterface</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">refreshIndividualRemoteSource</span><span style="color: #d4d4d4;">(</span><span style="color: #4ec9b0;">string</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceName</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">std</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">function</span><span style="color: #d4d4d4;"><</span><span style="color: #569cd6;">void</span><span style="color: #d4d4d4;">(</span><span style="color: #569cd6;">unsigned</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">progress</span><span style="color: #d4d4d4;">)></span><span style="color: #569cd6;">*</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">progressCallback</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;">{</span></div><div><span style="color: #6a9955;"> //cout << "Refreshing source " << remoteSourceName << endl << flush;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">InstallSource</span><span style="color: #d4d4d4;">* </span><span style="color: #9cdcfe;">source</span><span style="color: #d4d4d4;"> = </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #dcdcaa;">getRemoteSource</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">remoteSourceName</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">result</span><span style="color: #d4d4d4;"> = </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_installMgr</span><span style="color: #d4d4d4;">-></span><span style="color: #dcdcaa;">refreshRemoteSource</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">source</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> (</span><span style="color: #9cdcfe;">result</span><span style="color: #d4d4d4;"> != </span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">) {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">cerr</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #ce9178;">"Failed to refresh source "</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceName</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">endl</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">flush</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> }</span></div>
<div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceUpdateMutex</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">lock</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_remoteSourceUpdateCount</span><span style="color: #d4d4d4;">++;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">unsigned</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">totalPercent</span><span style="color: #d4d4d4;"> = (</span><span style="color: #569cd6;">unsigned</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;">)</span><span style="color: #dcdcaa;">calculateIntPercentage</span><span style="color: #d4d4d4;"><</span><span style="color: #569cd6;">double</span><span style="color: #d4d4d4;">>(</span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_remoteSourceUpdateCount</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_remoteSourceCount</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> (</span><span style="color: #9cdcfe;">progressCallback</span><span style="color: #d4d4d4;"> != </span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">) {</span></div><div><span style="color: #d4d4d4;"> (*</span><span style="color: #9cdcfe;">progressCallback</span><span style="color: #d4d4d4;">)</span><span style="color: #dcdcaa;">(</span><span style="color: #9cdcfe;">totalPercent</span><span style="color: #dcdcaa;">)</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceUpdateMutex</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">unlock</span><span style="color: #d4d4d4;">();</span></div>
<div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">return</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">result</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;">}</span></div></div>
<p>Best regards,<br>
Tobias<br>
</p>
<div class="moz-cite-prefix">On 10/12/20 9:01 PM, Troy A.
Griffitts wrote:<br>
</div>
<blockquote type="cite"
cite="mid:36F83DB3-F6C4-4D15-9146-40A6C5B762A0@crosswire.org">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Any luck getting a stack trace on crash?<br>
<br>
Regarding the "multitheaded mode", I'd have to get a bit
more information as to exactly how you are sharing SWORD
objects across your threads. Generally, as a rule, you
shouldn't. We recommend a separate instance of SWMgr per
thread and that probably goes for InstallMgr, as well.<br>
<br>
Troy<br>
<br>
<div class="gmail_quote">On October 12, 2020 8:29:31 PM
GMT+02:00, Tobias Klein <a class="moz-txt-link-rfc2396E"
href="mailto:contact@tklein.info" moz-do-not-send="true"><contact@tklein.info></a>
wrote:
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<p>Hi Troy,</p>
<p>I'm using curl on all three platforms.</p>
<p>Regarding the timeout configuration I have not
changed anything yet, to make this configurable in
Ezra Project is still on my todo list.</p>
<p>I just checked on Linux.<br>
With the old version (May 18th 2020) no hanging or
crash in 10 out of 10 times.<br>
WIth the new version (latest trunk / SWORD 1.9 RC3) I
get 1 x crash, 2 x hanging, 7 x working.</p>
<p>I'm running the InstallMgr::refreshRemoteSource "in a
multi-threaded mode".<br>
</p>
<p>Best regards,<br>
Tobias<br>
</p>
<div class="moz-cite-prefix">On 10/12/20 6:59 PM, Troy
A. Griffitts wrote:<br>
</div>
<blockquote type="cite"
cite="mid:E1689B41-6534-4EE4-9267-22BE9F59941E@crosswire.org">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
Hi Tobias,<br>
<br>
What transport library are you building with? ftplib
or curl?<br>
<br>
Have you changed the value of our new timeout from the
default, I believe we decided on, 10 seconds?<br>
<br>
Troy<br>
<br>
<div class="gmail_quote">On October 12, 2020 6:46:54
PM GMT+02:00, Tobias Klein <a
class="moz-txt-link-rfc2396E"
href="mailto:contact@tklein.info"
moz-do-not-send="true"><contact@tklein.info></a>
wrote:
<blockquote class="gmail_quote" style="margin: 0pt
0pt 0pt 0.8ex; border-left: 1px solid rgb(204,
204, 204); padding-left: 1ex;">
<pre class="k9mail">Hi Troy,
In my latest Ezra Project builds using SWORD trunk I’ve been noticing random „hangs“ and crashes related to "updating remote sources“. I suppose it must be around InstallMgr::refreshRemoteSource.
This was still rock solid when using SWORD trunk from May 18th 2020, but not so any more with the recent SWORD trunk.
Unfortunately I cannot pinpoint this more specifically. I just wanted to first share this observation, because it’s worrying me.
I’ve been noticing this regression both on Windows and macOS. Need to check later whether this also happens on Linux, cannot recall it right now.
Best regards,
Tobias<hr>sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
</div>
<br>
-- <br>
Sent from my Android device with K-9 Mail. Please
excuse my brevity. </blockquote>
</blockquote>
</div>
<br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my
brevity. </blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
</body>
</html>