<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>OK Tobias,</p>
<p>Give it a go when you have a chance and let me know.</p>
<p>Troy</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 10/17/20 12:16 PM, Troy A. Griffitts
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8C840408-9001-4524-82BC-67696C7B864E@crosswire.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
The unTarGZ is also a new method and it looks like it is using the
default file handle pool functionality of FileMgr, from looking at
your stack trace. Give me about an hour and I'll have a chance to
take a look at it. Good news is that it's not having trouble in
the CURLFTPTransport. I have the same change queued up for commit
for the other 3 transport impls so I will go ahead and commit
those, as well. Thank you for working through this with me.<br>
<br>
Troy<br>
<br>
<div class="gmail_quote">On October 17, 2020 10:44:18 AM
GMT+02:00, Tobias Klein <a class="moz-txt-link-rfc2396E" href="mailto:contact@tklein.info"><contact@tklein.info></a> wrote:
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<p>Dear Troy,</p>
<p>Thank you so much for the help and all your work on this.
Unfortunately the issue is still not resolved for me based
on your latest commits.<br>
</p>
<p>I have n threads that all run
InstallMgr::refreshRemoteSource. n corresponds to the number
of repositories available, so it's currently 10.<br>
The operation works until at some point things start hanging
again. Sometime it happens after ten consecutive calls of
the update function, sometimes already after three times and
I also had it hanging after only one attempt.<br>
In the calling function the hanging occurs when I join the
threads (waiting for them to complete).<br>
<br>
Looking at details in gdb I find this at the point of
hanging:</p>
<p><tt>(gdb) info threads</tt><tt><br>
</tt><tt> Id Target Id Frame </tt><tt><br>
</tt><tt> 102 Thread 0x7f8e3cdd1700 (LWP 46520)
"node_sword_cli" 0x000056411eacb46c in
sword::FileMgr::sysOpen(sword::FileDesc*) ()</tt><tt><br>
</tt><tt> 95 Thread 0x7f8e37fff700 (LWP 46514)
"node_sword_cli" 0x000056411eacb283 in
sword::FileMgr::sysOpen(sword::FileDesc*) ()</tt><tt><br>
</tt><tt> 93 Thread 0x7f8e3ddf6700 (LWP 46511)
"node_sword_cli" 0x000056411eacb283 in
sword::FileMgr::sysOpen(sword::FileDesc*) ()</tt><tt><br>
</tt><tt> 91 Thread 0x7f8e2bfff700 (LWP 46510)
"node_sword_cli" 0x000056411eacb46c in
sword::FileMgr::sysOpen(sword::FileDesc*) ()</tt><tt><br>
</tt><tt>* 90 Thread 0x7f8e2b7fe700 (LWP 46509)
"node_sword_cli" 0x000056411eacb296 in
sword::FileMgr::sysOpen(sword::FileDesc*) ()</tt><tt><br>
</tt><tt> 1 Thread 0x7f8e3ddf9e00 (LWP 46380)
"node_sword_cli" 0x00007f8e40d98cd7 in
__pthread_clockjoin_ex () from
/lib/x86_64-linux-gnu/libpthread.so.0</tt><tt><br>
</tt></p>
<p><tt>And this stacktrace for each individual thread
(relevant portion):</tt><tt><br>
</tt><tt>#0 0x000056411eacb46c in
sword::FileMgr::sysOpen(sword::FileDesc*) ()</tt><tt><br>
</tt><tt>#1 0x000056411eab4b0b in sword::FileDesc::getFd()
()</tt><tt><br>
</tt><tt>#2 0x000056411eb2fb70 in
sword::ZipCompress::unTarGZ(sword::FileDesc*, char const*)
()</tt><tt><br>
</tt><tt>#3 0x000056411eae4437 in
sword::InstallMgr::refreshRemoteSource(sword::InstallSource*)
()</tt><br>
</p>
<p>I'm not sure whether all of these threads here are now
hanging or only one of them. It could be just the one that
the main function tries to join right now.</p>
<p>Another observation is that I am getting random output like
this during the process (it happens with different conf
files, not this one all the time):</p>
<p><tt>error writing
/home/tobi/.sword/installMgr/20120711005000/mods.d/ngu_BL_1987.conf
skipping...</tt><br>
<br>
I didn't get these error messages with earlier SVN
revisions.<br>
</p>
<p>To be sure I just once more tested with SVN Rev. 3759 and
there I consistently get 20 out of 20 attempts successful.<br>
</p>
Best regards,<br>
Tobias<br>
<br>
PS: I'm sending this e-mail the second time, didn't seem to
come through via mailman the first time (at 9:11 CEST).<br>
<br>
<div class="moz-cite-prefix">On 10/15/20 8:09 PM, Troy A.
Griffitts wrote:<br>
</div>
<blockquote type="cite"
cite="mid:5a02225c-4948-818d-8a0d-b0b8c3310835@crosswire.org">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>Dear Tobias,</p>
<p>Thank you for all the great information. This enabled me
to isolate the change which caused the issue.</p>
<p>So, for a bit of background, SWORD has no calls to mark
critical sections which might be problematic for
re-entrant usage. This has been due to the many
implementations of threading across many different
platforms over the years, before C++11. But, as a policy
to support clients which desire to use SWORD in a
multithreaded manner, we do our best to make this safe by
advising clients to use separate SWMgr instances per
thread. There are still some shared objects in this
scenario, but we do our best to do all the writing to
these shared objects upon initialization. We broke this
rule in commit 2760, which is what caused your problem.
SWORD have a facility to pool open file handles, to help
OSs which have small open file handle limits. This work
is done in FileMgr. Recently, to support Windows Unicode
path names (the commit you found which breaks your
multithreaded use), we rounded up all remaining native
file IO calls and replaced them to used FileMgr for the IO
and then extended FileMgr to handle Windows Unicode paths
in a Windows-specific manner. One of these changes was in
CURLFTPTransport, which is where you are having the
issue. The problem is that, where previously this class
was directly opening a FILE to do its writing, commit 2760
changed this to use FileMgr to open the file, which
involved the SWORD-wide file handle pool, and since we are
create a new file, we are always writing to this shared
pool container, which is not threadsafe. My guess is that
you have two threads trying to update the pool container
at exactly the same time. Using the file handle pool is
usually safe, because SWMgr "opens" all of its file
handles on initialization (these are not actually opening
OS file handles, but instead updating the file handle pool
container with proxy objects which delay actual OS open to
on-demand, but the point is this instance of shared file
handle pool container writing is done on creation of the
SWMgr, afterward, the shared resource file handle pool is
only read and each object in the pool is owned by only 1
thread if the "each thread must have its own SWMgr" rule
is followed.<br>
</p>
<p>Regardless of the details. I believe I have committed a
fix for you. In short, I have changed CURLFTPTransport to
follow our rule to avoid writing to shared objects when we
might be re-entrant. Here we now use FileMgr's methods
which isolate OS implementation, but not FileMgr's file
handle pool (as it did not previously use the pool before
this commit). This should allow this to still take
advantage of the Windows OS-specific implementation, and
also avoid the critical section. Can you please try SVN
head and let me know if we are back to 20 out of 20
successes?</p>
<p>Thanks again for the very helpful debug log and exact
revision where failure began.</p>
<p>Troy<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 10/13/20 10:08 PM, Tobias
Klein wrote:<br>
</div>
<blockquote type="cite"
cite="mid:3258796d-a77a-0009-b0b3-821cda1c89a4@tklein.info">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>I managed to get a backtrace to a segmentation fault
using GDB.</p>
<p>It seems like the crash is happening in
sword::FileMgr::open( ...</p>
<p>The starting point is
sword::InstallMgr::refreshRemoteSource as I was writing
before.</p>
<p>Best regards,<br>
Tobias<br>
</p>
<p>Program received signal SIGSEGV, Segmentation fault.<br>
[Switching to Thread 0x7f1af3fff700 (LWP 220833)]<br>
0x00007f1b027045a4 in sword::FileMgr::open(char const*,
int, int, bool) () from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
(gdb) backtrace<br>
#0 0x00007f1b027045a4 in sword::FileMgr::open(char
const*, int, int, bool) () from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#1 0x00007f1b0276ad7b in sword::(anonymous
namespace)::my_fwrite(void*, unsigned long, unsigned
long, void*) ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#2 0x00007f1b180626bf in ?? () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#3 0x00007f1b18074a2b in ?? () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#4 0x00007f1b1807e2e4 in ?? () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#5 0x00007f1b1807f6f9 in curl_multi_perform () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#6 0x00007f1b18075d13 in curl_easy_perform () from
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4<br>
#7 0x00007f1b0276b683 in
sword::CURLFTPTransport::getURL(char const*, char
const*, sword::SWBuf*) () from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#8 0x00007f1b0271d5d2 in
sword::InstallMgr::remoteCopy(sword::InstallSource*,
char const*, char const*, bool, char const*) ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#9 0x00007f1b0271edc7 in
sword::InstallMgr::refreshRemoteSource(sword::InstallSource*)
() from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#10 0x00007f1b026ad734 in
RepositoryInterface::refreshIndividualRemoteSource(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char>
>, std::function<void (unsigned int)>*) ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#11 0x00007f1b026b17dd in
std::thread::_State_impl<std::thread::_Invoker<std::tuple<int
(RepositoryInterface::*)(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char>
>, std::function<void (unsigned int)>*),
RepositoryInterface*,
std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char>
>, std::function<void (unsigned int)>*> >
>::_M_run() ()<br>
from
/home/tobi/dev/ezra_project/node-sword-interface-git/build/Release/node_sword_interface.node<br>
#12 0x00007f1b1d622cb4 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6<br>
#13 0x00007f1b1e20a609 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0<br>
#14 0x00007f1b1e131103 in clone () from
/lib/x86_64-linux-gnu/libc.so.6<br>
</p>
<div class="moz-cite-prefix">On 10/13/20 1:07 PM, Tobias
Klein wrote:<br>
</div>
<blockquote type="cite"
cite="mid:03bca06c-8c98-434a-ff32-3947e285961a@tklein.info">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>Hi Troy,</p>
<p>I tested more SVN revisions of SWORD trunk (starting
from my stable version until I hit the bug) and I can
now say that</p>
<p>SVN Rev. 3759 is the last SVN revision that works
without hanging for the below mentioned scenario. (20
out of 20 tests successful)<br>
</p>
<p>SVN Rev. 3760 is the first SVN revision where the
hanging occurs. The commit message is "First cut at
better isolation of FileIO to FileMgr and providing a
WIN32 impl with works with wchar_t".</p>
<p>Modified files:<br>
include/filemgr.h<br>
include/swbuf.h<br>
lib/bcppmake/libsword.bpr<br>
src/mgr/curlftpt.cpp<br>
src/mgr/curlhttpt.cpp<br>
src/mgr/filemgr.cpp<br>
src/mgr/installmgr.cpp<br>
src/mgr/swmgr.cpp<br>
src/utilfuns/utilstr.cpp<br>
<br>
Maybe this helps to find the root-cause.<br>
<br>
Best regards,<br>
Tobias<br>
</p>
<div class="moz-cite-prefix">On 10/12/20 9:20 PM, Tobias
Klein wrote:<br>
</div>
<blockquote type="cite"
cite="mid:fcc4672a-feaf-4eec-9f7c-a0acdc57faf0@tklein.info">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>I'll see whether I can collect a stack trace. It
may take some time until I have it.</p>
<p>The multi-threaded "remote source refreshing"
worked without issues until recently.<br>
</p>
<p>Here is the code of the function that does the
actual work in a thread.<br>
See <a class="moz-txt-link-freetext"
href="https://github.com/tobias-klein/node-sword-interface/blob/787160ccb4b3bab2a762d22f74031c7237edc803/src/sword_backend/repository_interface.cpp#L105"
moz-do-not-send="true">https://github.com/tobias-klein/node-sword-interface/blob/787160ccb4b3bab2a762d22f74031c7237edc803/src/sword_backend/repository_interface.cpp#L105</a>.</p>
<div style="color: #d4d4d4;background-color: #1e1e1e;font-family: 'Droid Sans Mono', 'monospace', monospace, 'Droid Sans Fallback';font-weight: normal;font-size: 14px;line-height: 19px;white-space: pre;"><div><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">RepositoryInterface</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">refreshIndividualRemoteSource</span><span style="color: #d4d4d4;">(</span><span style="color: #4ec9b0;">string</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceName</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">std</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">function</span><span style="color: #d4d4d4;"><</span><span style="color: #569cd6;">void</span><span style="color: #d4d4d4;">(</span><span style="color: #569cd6;">unsigned</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">progress</span><span style="color: #d4d4d4;">)></span><span style="color: #569cd6;">*</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">progressCallback</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;">{</span></div><div><span style="color: #6a9955;"> //cout << "Refreshing source " << remoteSourceName << endl << flush;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">InstallSource</span><span style="color: #d4d4d4;">* </span><span style="color: #9cdcfe;">source</span><span style="color: #d4d4d4;"> = </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #dcdcaa;">getRemoteSource</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">remoteSourceName</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">result</span><span style="color: #d4d4d4;"> = </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_installMgr</span><span style="color: #d4d4d4;">-></span><span style="color: #dcdcaa;">refreshRemoteSource</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">source</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> (</span><span style="color: #9cdcfe;">result</span><span style="color: #d4d4d4;"> != </span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">) {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">cerr</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #ce9178;">"Failed to refresh source "</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceName</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">endl</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;"><<</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">flush</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> }</span></div>
<div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceUpdateMutex</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">lock</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_remoteSourceUpdateCount</span><span style="color: #d4d4d4;">++;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">unsigned</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">totalPercent</span><span style="color: #d4d4d4;"> = (</span><span style="color: #569cd6;">unsigned</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">int</span><span style="color: #d4d4d4;">)</span><span style="color: #dcdcaa;">calculateIntPercentage</span><span style="color: #d4d4d4;"><</span><span style="color: #569cd6;">double</span><span style="color: #d4d4d4;">>(</span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_remoteSourceUpdateCount</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">this</span><span style="color: #d4d4d4;">-></span><span style="color: #9cdcfe;">_remoteSourceCount</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> (</span><span style="color: #9cdcfe;">progressCallback</span><span style="color: #d4d4d4;"> != </span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">) {</span></div><div><span style="color: #d4d4d4;"> (*</span><span style="color: #9cdcfe;">progressCallback</span><span style="color: #d4d4d4;">)</span><span style="color: #dcdcaa;">(</span><span style="color: #9cdcfe;">totalPercent</span><span style="color: #dcdcaa;">)</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">remoteSourceUpdateMutex</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">unlock</span><span style="color: #d4d4d4;">();</span></div>
<div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">return</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">result</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;">}</span></div></div>
<p>Best regards,<br>
Tobias<br>
</p>
<div class="moz-cite-prefix">On 10/12/20 9:01 PM, Troy
A. Griffitts wrote:<br>
</div>
<blockquote type="cite"
cite="mid:36F83DB3-F6C4-4D15-9146-40A6C5B762A0@crosswire.org">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Any luck getting a stack trace on crash?<br>
<br>
Regarding the "multitheaded mode", I'd have to get a
bit more information as to exactly how you are
sharing SWORD objects across your threads.
Generally, as a rule, you shouldn't. We recommend a
separate instance of SWMgr per thread and that
probably goes for InstallMgr, as well.<br>
<br>
Troy<br>
<br>
<div class="gmail_quote">On October 12, 2020 8:29:31
PM GMT+02:00, Tobias Klein <a
class="moz-txt-link-rfc2396E"
href="mailto:contact@tklein.info"
moz-do-not-send="true"><contact@tklein.info></a>
wrote:
<blockquote class="gmail_quote" style="margin: 0pt
0pt 0pt 0.8ex; border-left: 1px solid rgb(204,
204, 204); padding-left: 1ex;">
<p>Hi Troy,</p>
<p>I'm using curl on all three platforms.</p>
<p>Regarding the timeout configuration I have
not changed anything yet, to make this
configurable in Ezra Project is still on my
todo list.</p>
<p>I just checked on Linux.<br>
With the old version (May 18th 2020) no
hanging or crash in 10 out of 10 times.<br>
WIth the new version (latest trunk / SWORD 1.9
RC3) I get 1 x crash, 2 x hanging, 7 x
working.</p>
<p>I'm running the
InstallMgr::refreshRemoteSource "in a
multi-threaded mode".<br>
</p>
<p>Best regards,<br>
Tobias<br>
</p>
<div class="moz-cite-prefix">On 10/12/20 6:59
PM, Troy A. Griffitts wrote:<br>
</div>
<blockquote type="cite"
cite="mid:E1689B41-6534-4EE4-9267-22BE9F59941E@crosswire.org">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
Hi Tobias,<br>
<br>
What transport library are you building with?
ftplib or curl?<br>
<br>
Have you changed the value of our new timeout
from the default, I believe we decided on, 10
seconds?<br>
<br>
Troy<br>
<br>
<div class="gmail_quote">On October 12, 2020
6:46:54 PM GMT+02:00, Tobias Klein <a
class="moz-txt-link-rfc2396E"
href="mailto:contact@tklein.info"
moz-do-not-send="true"><contact@tklein.info></a>
wrote:
<blockquote class="gmail_quote"
style="margin: 0pt 0pt 0pt 0.8ex;
border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<pre class="k9mail">Hi Troy,
In my latest Ezra Project builds using SWORD trunk I’ve been noticing random „hangs“ and crashes related to "updating remote sources“. I suppose it must be around InstallMgr::refreshRemoteSource.
This was still rock solid when using SWORD trunk from May 18th 2020, but not so any more with the recent SWORD trunk.
Unfortunately I cannot pinpoint this more specifically. I just wanted to first share this observation, because it’s worrying me.
I’ve been noticing this regression both on Windows and macOS. Need to check later whether this also happens on Linux, cannot recall it right now.
Best regards,
Tobias<hr>sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
</div>
<br>
-- <br>
Sent from my Android device with K-9 Mail.
Please excuse my brevity. </blockquote>
</blockquote>
</div>
<br>
-- <br>
Sent from my Android device with K-9 Mail. Please
excuse my brevity. </blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org" moz-do-not-send="true">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel" moz-do-not-send="true">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
</blockquote>
</div>
<br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my
brevity.
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://crosswire.org/mailman/listinfo/sword-devel">http://crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
</body>
</html>