<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thanks for investigating this DM. I had a look today and
observed the same behavior. Watching: `tail -f
crosswire-access_log|grep study` I see almost exclusively bots, a
couple requests each second. We have a robots.txt file which
should prevent them from crawling swordweb (sadly... it would be
nice if this was searchable in whatever database these bots are
populating) but they don't see to be obeying.</p>
<p>My attempt to remediate:<br>
<br>
I've added a robots.txt entry for the entire top level study/
folder and also a `Crawl-delay: 30`. We'll see if they honor any
of that.</p>
<p><a class="moz-txt-link-freetext" href="https://crosswire.org/robots.txt">https://crosswire.org/robots.txt</a></p>
<p>I've changed the tomcat session timeout to 7 minutes (from 30
minutes), which will expire the swordorbserver instances from bots
much faster.</p>
<p>/home/swordweb/servers/main/conf/web.xml#L582<br>
</p>
<p>We may need to start blacklisting agents in fail2ban or spend a
bit of time rethinking the TTL for swordorbserver. It doesn't
hurt to nicely `killall swordorbserver`. If a session can't
connect to its orb, it will spawn a new one-- it just takes the
extra startup time for sword to read its library, which is pretty
quick on our server anyway. The issue with bots is that
swordorbserver spawns per session, and bots never persist a
session so a new session is opened per request.</p>
<p>Let's see how this does,</p>
<p>Troy</p>
<p> <br>
</p>
<div class="moz-cite-prefix">On 9/11/25 2:01 AM, DM Smith wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1FF90B9B-CA34-4FD4-8D12-061ABAAB7AE7@crosswire.org">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
This time I didn’t fall asleep. The culprit is the swordorbserver
processes. There were 8-10 created every 10 seconds. I created a
loop to spit out the count every 10 seconds. The last count before
jira was killed was 1040.
<div><br>
</div>
<div><br>
<div>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">top -
19:56:20 up 4 days, 4:46, 2 users, load average: 1.07,
1.06, 1.41</span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">Tasks:<b>
1504 </b>total,<b> 2 </b>running,<b> 1502 </b>sleeping,<b>
0 </b>stopped,<b> 0 </b>zombie</span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">%Cpu(s):<b>
3.8 </b>us,<b> 3.1 </b>sy,<b> 0.0 </b>ni,<b> 92.5 </b>id,<b>
0.4 </b>wa,<b> 0.0 </b>hi,<b> 0.1 </b>si,<b> 0.0 </b>st</span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">MiB
Mem :<b> 78860.2 </b>total,<b> 505.7 </b>free,<b>
78156.8 </b>used,<b> 197.8 </b>buff/cache</span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">MiB
Swap:<b> 4883.0 </b>total,<b> 0.0 </b>free,<b>
4883.0 </b>used.<b> 56.6 </b>avail Mem </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 21px;"><span
style="font-variant-ligatures: no-common-ligatures"></span><br>
</p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(255, 255, 255);"><span
style="font-variant-ligatures: no-common-ligatures; background-color: #000000">
PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2939079
jira 20 0 13.1g 1.3g 5804 S 0.0 1.7
3:42.35 java </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2941639
swordweb 20 0 22.0g 578728 5796 S 18.5 0.7
1:35.40 java </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2765973
tomcat 20 0 11.0g 229580 3744 S 0.3 0.3
1:07.09 java </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">
1886 mysql 20 0 8814624 204424 1972 S 6.6 0.3
255:40.49 mariadbd </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2930053
crosswi+ 20 0 25.5g 107332 6924 S 0.3 0.1
0:28.18 java </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">
1530 vmrcre 20 0 18.9g 97908 0 S 0.3 0.1
70:37.28 java </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2949492
swordweb 20 0 369652 92396 13660 S 0.0 0.1
0:00.43 swordorbserver </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2949512
swordweb 20 0 370200 91440 13508 S 0.0 0.1
0:00.45 swordorbserver </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2949528
swordweb 20 0 370200 91348 13420 S 0.0 0.1
0:00.44 swordorbserver </span></p>
<p
style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span
style="font-variant-ligatures: no-common-ligatures">2949409
swordweb 20 0 370200 91336 13628 S 0.0 0.1
0:00.46 swordorbserver </span></p>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures: no-common-ligatures"><br>
</span></div>
<div><br>
<blockquote type="cite">
<div>On Sep 10, 2025, at 7:34 PM, DM Smith
<a class="moz-txt-link-rfc2396E" href="mailto:dmsmith@crosswire.org"><dmsmith@crosswire.org></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
<div
style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">I
watched “top” sorted by RSS and Jira was at the top.
The RSS slowly went up to 1.5G and then 1.6 and
finally to 1.7. But processes went from 500 or so to
over 1200, when I fell asleep watching it die a third
time. Lisa alerted me that I had fallen asleep!
<div><br>
</div>
<div>I noticed that 3 of the top processes, all java,
were killed. 2 restarted (monit?). The number of
processes dropped to around 500 and have been
creeping upward and it’s nearly 900 now.</div>
<div><br>
</div>
<div>I continued to watch and swordweb died and
restarted again.</div>
<div><br>
</div>
<div>Hope this helps.</div>
<div><br>
</div>
<div>DM<br>
<div><br>
<div>
<blockquote type="cite">
<div>On Sep 10, 2025, at 4:47 PM, DM Smith
<a class="moz-txt-link-rfc2396E" href="mailto:dmsmith@crosswire.org"><dmsmith@crosswire.org></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
<div
style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">And
it died again after a few minutes. I
restarted it. Not hopeful.
<div><br>
</div>
<div>— DM<br
id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Sep 10, 2025, at 4:17 PM, DM
Smith
<a class="moz-txt-link-rfc2396E" href="mailto:dmsmith@crosswire.org"><dmsmith@crosswire.org></a>
wrote:</div>
<br
class="Apple-interchange-newline">
<div>
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
<div
style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">I
don’t know how to triage or fix
the underlying problem. I’ve
restarted it.
<div><br>
</div>
<div>Looking at
/var/log/messages, it is a
machine OOM. Same as what Troy
saw before. I’m guessing that
the “OOM Reaper” is picking
the biggest memory hogs and
killing them. Those are all
java processes.</div>
<div>
<div><br>
</div>
<div>There are many, many
(~200) sword observer
processes each taking a
paltry 90,000 KB. Perhaps
these are the culprits? Is
there a bound on the pool of
sword observers?</div>
<div><br>
</div>
<div>I also noted that:</div>
<div>When the large java
process is killed (which
belongs to Jira) that many
mariadb connections by jira
are terminated.</div>
<div>Jira needs to be updated
from its current outdated
version. Perhaps the newer
version has a better memory
footprint?</div>
<div><br>
</div>
<div>— DM<br
id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Sep 10, 2025, at
8:11 AM, Karl
Kleinpaste
<a class="moz-txt-link-rfc2396E" href="mailto:karl@kleinpaste.org"><karl@kleinpaste.org></a>
wrote:</div>
<br
class="Apple-interchange-newline">
<div>
<meta
http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div>
<div
class="moz-cite-prefix">On 9/9/25 3:57 PM, DM Smith wrote:<br>
</div>
<blockquote
type="cite"
cite="mid:566A2789-8A0C-4AB6-882A-480D4CB00796@crosswire.org">I
restarted it.</blockquote>
<br>
<font
face="FreeSerif">And
it's dead again.<br>
<br>
Something is
evidently more
seriously wrong
than a mere need
to restart.<br>
</font> </div>
_______________________________________________<br>
server-admins mailing
list<br>
<a class="moz-txt-link-abbreviated" href="mailto:server-admins@crosswire.org">server-admins@crosswire.org</a><br>
<a class="moz-txt-link-freetext" href="https://crosswire.org/mailman/listinfo/server-admins">https://crosswire.org/mailman/listinfo/server-admins</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
_______________________________________________<br>
server-admins mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:server-admins@crosswire.org">server-admins@crosswire.org</a><br>
<a class="moz-txt-link-freetext" href="https://crosswire.org/mailman/listinfo/server-admins">https://crosswire.org/mailman/listinfo/server-admins</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
_______________________________________________<br>
server-admins mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:server-admins@crosswire.org">server-admins@crosswire.org</a><br>
<a class="moz-txt-link-freetext" href="https://crosswire.org/mailman/listinfo/server-admins">https://crosswire.org/mailman/listinfo/server-admins</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
_______________________________________________<br>
server-admins mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:server-admins@crosswire.org">server-admins@crosswire.org</a><br>
<a class="moz-txt-link-freetext" href="https://crosswire.org/mailman/listinfo/server-admins">https://crosswire.org/mailman/listinfo/server-admins</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre wrap="" class="moz-quote-pre">_______________________________________________
server-admins mailing list
<a class="moz-txt-link-abbreviated" href="mailto:server-admins@crosswire.org">server-admins@crosswire.org</a>
<a class="moz-txt-link-freetext" href="https://crosswire.org/mailman/listinfo/server-admins">https://crosswire.org/mailman/listinfo/server-admins</a>
</pre>
</blockquote>
</body>
</html>