[server-admins] MonIt

[DM Smith]

I watched “top” sorted by RSS and Jira was at the top. The RSS slowly went up to 1.5G and then 1.6 and finally to 1.7. But processes went from 500 or so to over 1200, when I fell asleep watching it die a third time. Lisa alerted me that I had fallen asleep!

I noticed that 3 of the top processes, all java, were killed. 2 restarted (monit?). The number of processes dropped to around 500 and have been creeping upward and it’s nearly 900 now.

I continued to watch and swordweb died and restarted again.

Hope this helps.

DM

> On Sep 10, 2025, at 4:47 PM, DM Smith <dmsmith at crosswire.org> wrote:
> 
> And it died again after a few minutes. I restarted it. Not hopeful.
> 
> — DM
> 
>> On Sep 10, 2025, at 4:17 PM, DM Smith <dmsmith at crosswire.org> wrote:
>> 
>> I don’t know how to triage or fix the underlying problem. I’ve restarted it.
>> 
>> Looking at /var/log/messages, it is a machine OOM. Same as what Troy saw before. I’m guessing that the “OOM Reaper” is picking the biggest memory hogs and killing them. Those are all java processes.
>> 
>> There are many, many (~200) sword observer processes each taking a paltry 90,000 KB. Perhaps these are the culprits? Is there a bound on the pool of sword observers?
>> 
>> I also noted that:
>> When the large java process is killed (which belongs to Jira) that many mariadb connections by jira are terminated.
>> Jira needs to be updated from its current outdated version. Perhaps the newer version has a better memory footprint?
>> 
>> — DM
>> 
>>> On Sep 10, 2025, at 8:11 AM, Karl Kleinpaste <karl at kleinpaste.org> wrote:
>>> 
>>> On 9/9/25 3:57 PM, DM Smith wrote:
>>>> I restarted it.
>>> 
>>> And it's dead again.
>>> 
>>> Something is evidently more seriously wrong than a mere need to restart.
>>> _______________________________________________
>>> server-admins mailing list
>>> server-admins at crosswire.org
>>> https://crosswire.org/mailman/listinfo/server-admins
>> 
>> _______________________________________________
>> server-admins mailing list
>> server-admins at crosswire.org
>> https://crosswire.org/mailman/listinfo/server-admins
> 
> _______________________________________________
> server-admins mailing list
> server-admins at crosswire.org
> https://crosswire.org/mailman/listinfo/server-admins

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://crosswire.org/pipermail/server-admins/attachments/20250910/237ed9e6/attachment.htm>