<html aria-label="message body"><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">This time I didn’t fall asleep. The culprit is the swordorbserver processes. There were 8-10 created every 10 seconds. I created a loop to spit out the count every 10 seconds. The last count before jira was killed was 1040.<div><br></div><div><br><div><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">top - 19:56:20 up 4 days, 4:46, 2 users, load average: 1.07, 1.06, 1.41</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">Tasks:<b> 1504 </b>total,<b> 2 </b>running,<b> 1502 </b>sleeping,<b> 0 </b>stopped,<b> 0 </b>zombie</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">%Cpu(s):<b> 3.8 </b>us,<b> 3.1 </b>sy,<b> 0.0 </b>ni,<b> 92.5 </b>id,<b> 0.4 </b>wa,<b> 0.0 </b>hi,<b> 0.1 </b>si,<b> 0.0 </b>st</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">MiB Mem :<b> 78860.2 </b>total,<b> 505.7 </b>free,<b> 78156.8 </b>used,<b> 197.8 </b>buff/cache</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">MiB Swap:<b> 4883.0 </b>total,<b> 0.0 </b>free,<b> 4883.0 </b>used.<b> 56.6 </b>avail Mem </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 21px;"><span style="font-variant-ligatures: no-common-ligatures"></span><br></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(255, 255, 255);"><span style="font-variant-ligatures: no-common-ligatures; background-color: #000000"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2939079 jira 20 0 13.1g 1.3g 5804 S 0.0 1.7 3:42.35 java </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2941639 swordweb 20 0 22.0g 578728 5796 S 18.5 0.7 1:35.40 java </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2765973 tomcat 20 0 11.0g 229580 3744 S 0.3 0.3 1:07.09 java </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures"> 1886 mysql 20 0 8814624 204424 1972 S 6.6 0.3 255:40.49 mariadbd </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2930053 crosswi+ 20 0 25.5g 107332 6924 S 0.3 0.1 0:28.18 java </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures"> 1530 vmrcre 20 0 18.9g 97908 0 S 0.3 0.1 70:37.28 java </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2949492 swordweb 20 0 369652 92396 13660 S 0.0 0.1 0:00.43 swordorbserver </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2949512 swordweb 20 0 370200 91440 13508 S 0.0 0.1 0:00.45 swordorbserver </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2949528 swordweb 20 0 370200 91348 13420 S 0.0 0.1 0:00.44 swordorbserver </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">2949409 swordweb 20 0 370200 91336 13628 S 0.0 0.1 0:00.46 swordorbserver </span></p><div><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div><br><blockquote type="cite"><div>On Sep 10, 2025, at 7:34 PM, DM Smith <dmsmith@crosswire.org> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">I watched “top” sorted by RSS and Jira was at the top. The RSS slowly went up to 1.5G and then 1.6 and finally to 1.7. But processes went from 500 or so to over 1200, when I fell asleep watching it die a third time. Lisa alerted me that I had fallen asleep!<div><br></div><div>I noticed that 3 of the top processes, all java, were killed. 2 restarted (monit?). The number of processes dropped to around 500 and have been creeping upward and it’s nearly 900 now.</div><div><br></div><div>I continued to watch and swordweb died and restarted again.</div><div><br></div><div>Hope this helps.</div><div><br></div><div>DM<br><div><br><div><blockquote type="cite"><div>On Sep 10, 2025, at 4:47 PM, DM Smith <dmsmith@crosswire.org> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">And it died again after a few minutes. I restarted it. Not hopeful.<div><br></div><div>— DM<br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Sep 10, 2025, at 4:17 PM, DM Smith <dmsmith@crosswire.org> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">I don’t know how to triage or fix the underlying problem. I’ve restarted it.<div><br></div><div>Looking at /var/log/messages, it is a machine OOM. Same as what Troy saw before. I’m guessing that the “OOM Reaper” is picking the biggest memory hogs and killing them. Those are all java processes.</div><div><div><br></div><div>There are many, many (~200) sword observer processes each taking a paltry 90,000 KB. Perhaps these are the culprits? Is there a bound on the pool of sword observers?</div><div><br></div><div>I also noted that:</div><div>When the large java process is killed (which belongs to Jira) that many mariadb connections by jira are terminated.</div><div>Jira needs to be updated from its current outdated version. Perhaps the newer version has a better memory footprint?</div><div><br></div><div>— DM<br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Sep 10, 2025, at 8:11 AM, Karl Kleinpaste <karl@kleinpaste.org> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div>
<div class="moz-cite-prefix">On 9/9/25 3:57 PM, DM Smith wrote:<br>
</div>
<blockquote type="cite" cite="mid:566A2789-8A0C-4AB6-882A-480D4CB00796@crosswire.org">I
restarted it.</blockquote>
<br>
<font face="FreeSerif">And it's dead again.<br>
<br>
Something is evidently more seriously wrong than a mere need to
restart.<br>
</font>
</div>
_______________________________________________<br>server-admins mailing list<br>server-admins@crosswire.org<br>https://crosswire.org/mailman/listinfo/server-admins<br></div></blockquote></div><br></div></div></div>_______________________________________________<br>server-admins mailing list<br>server-admins@crosswire.org<br>https://crosswire.org/mailman/listinfo/server-admins<br></div></blockquote></div><br></div></div>_______________________________________________<br>server-admins mailing list<br>server-admins@crosswire.org<br>https://crosswire.org/mailman/listinfo/server-admins<br></div></blockquote></div><br></div></div></div>_______________________________________________<br>server-admins mailing list<br>server-admins@crosswire.org<br>https://crosswire.org/mailman/listinfo/server-admins<br></div></blockquote></div><br></div></div></body></html>