[server-admins] MonIt

[DM Smith]

Yes, Praise Jesus

and thank you so much!



> On Aug 13, 2025, at 11:34 PM, Troy A. Griffitts <scribe at crosswire.org> wrote:
> 
> OK, so I did a few things tonight.  I upgraded our hypervisor from RHEL 8 -> 9.  I updated its ancient dell firmware which was all about 5 years old.  Had a few hiccups with the firmware upgrade but prayed a lot and we're booted up again, praise Jesus.  The RHEL upgrade went pretty smooth.  The VMs wouldn't boot because they used something called SPLICE or something similar and had to be converted to whatever is not SPLICE but the rhel cockpit web console prompted me to click a button for the upgrade and all went well.  I've bumped the crosswire.org guest VM +16GB of RAM.  Let's see how things run with the extra space.
> 
> Praise Jesus all is OK!
> 
> Troy
> 
> 
> 
> On 8/13/25 11:57 PM, Troy A. Griffitts wrote:
>> Well, hunting through /var/log/messages I see:
>> 
>> Aug 13 12:32:22 host kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/jira.service,task=java,pid=2664516,uid=1008
>> Aug 13 12:32:22 host kernel: Out of memory: Killed process 2664516 (java) total-vm:14400952kB, anon-rss:1966876kB, file-rss:0kB, shmem-rss:0kB, UID:1008 pgtables:5156kB oom_score_adj:0
>> Aug 13 12:32:22 host kernel: oom_reaper: reaped process 2664516 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6770954 [Warning] Aborted connection 6770954 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6770681 [Warning] Aborted connection 6770681 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6770942 [Warning] Aborted connection 6770942 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6824722 [Warning] Aborted connection 6824722 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6828258 [Warning] Aborted connection 6828258 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6838939 [Warning] Aborted connection 6838939 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6824721 [Warning] Aborted connection 6824721 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6831922 [Warning] Aborted connection 6831922 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6821712 [Warning] Aborted connection 6821712 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6821711 [Warning] Aborted connection 6821711 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6828256 [Warning] Aborted connection 6828256 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6824720 [Warning] Aborted connection 6824720 to db: 'jiradb7' user: 'jiradbuser' host: 'localhost' (Got an error reading communication packets)
>> Aug 13 12:32:22 host systemd[1]: jira.service: Main process exited, code=killed, status=9/KILL
>> 
>> I dont know if those db errors are due to the processes being killed or if they might be a cause of the oom.
>> 
>> Still a mystery why the oom.  If we need more memory, I can allocate more to our guest, which has 64GB right now and using half of that currently, but jira is down.
>> 
>> Troy
>> 
>> 
>> 
>> On 8/13/25 3:47 PM, Karl Kleinpaste wrote:
>>> On 8/12/25 7:25 PM, DM Smith wrote:
>>>> Well it clearly isn’t Monit’s fault.
>>> 
>>> I don't know whose fault it is, but Jira is AWOL right now.
>>> 
>>> 
>>> _______________________________________________
>>> server-admins mailing list
>>> server-admins at crosswire.org <mailto:server-admins at crosswire.org>
>>> http://www.crosswire.org/mailman/listinfo/server-admins
>> 
>> 
>> _______________________________________________
>> server-admins mailing list
>> server-admins at crosswire.org <mailto:server-admins at crosswire.org>
>> http://www.crosswire.org/mailman/listinfo/server-admins
> _______________________________________________
> server-admins mailing list
> server-admins at crosswire.org
> http://www.crosswire.org/mailman/listinfo/server-admins

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/server-admins/attachments/20250813/8f51d580/attachment-0001.htm>