[server-admins] MonIt

[Troy A. Griffitts]

OK, so I did a few things tonight.  I upgraded our hypervisor from RHEL 
8 -> 9.  I updated its ancient dell firmware which was all about 5 years 
old.  Had a few hiccups with the firmware upgrade but prayed a lot and 
we're booted up again, praise Jesus. The RHEL upgrade went pretty 
smooth.  The VMs wouldn't boot because they used something called SPLICE 
or something similar and had to be converted to whatever is not SPLICE 
but the rhel cockpit web console prompted me to click a button for the 
upgrade and all went well.  I've bumped the crosswire.org guest VM +16GB 
of RAM. Let's see how things run with the extra space.

Praise Jesus all is OK!

Troy


On 8/13/25 11:57 PM, Troy A. Griffitts wrote:
>
> Well, hunting through /var/log/messages I see:
>
> Aug 13 12:32:22 host kernel: 
> oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/jira.service,task=java,pid=2664516,uid=1008
> Aug 13 12:32:22 host kernel: Out of memory: Killed process 2664516 
> (java) total-vm:14400952kB, anon-rss:1966876kB, file-rss:0kB, 
> shmem-rss:0kB, UID:1008 pgtables:5156kB oom_score_adj:0
> Aug 13 12:32:22 host kernel: oom_reaper: reaped process 2664516 
> (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6770954 
> [Warning] Aborted connection 6770954 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6770681 
> [Warning] Aborted connection 6770681 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6770942 
> [Warning] Aborted connection 6770942 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6824722 
> [Warning] Aborted connection 6824722 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6828258 
> [Warning] Aborted connection 6828258 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6838939 
> [Warning] Aborted connection 6838939 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6824721 
> [Warning] Aborted connection 6824721 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6831922 
> [Warning] Aborted connection 6831922 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6821712 
> [Warning] Aborted connection 6821712 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6821711 
> [Warning] Aborted connection 6821711 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6828256 
> [Warning] Aborted connection 6828256 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host mariadbd[1819]: 2025-08-13 12:32:22 6824720 
> [Warning] Aborted connection 6824720 to db: 'jiradb7' user: 
> 'jiradbuser' host: 'localhost' (Got an error reading communication 
> packets)
> Aug 13 12:32:22 host systemd[1]: jira.service: Main process exited, 
> code=killed, status=9/KILL
>
> I dont know if those db errors are due to the processes being killed 
> or if they might be a cause of the oom.
>
> Still a mystery why the oom.  If we need more memory, I can allocate 
> more to our guest, which has 64GB right now and using half of that 
> currently, but jira is down.
>
> Troy
>
>
> On 8/13/25 3:47 PM, Karl Kleinpaste wrote:
>> On 8/12/25 7:25 PM, DM Smith wrote:
>>> Well it clearly isn’t Monit’s fault.
>>
>> I don't know whose fault it is, but Jira is AWOL right now.
>>
>> _______________________________________________
>> server-admins mailing list
>> server-admins at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/server-admins
>
> _______________________________________________
> server-admins mailing list
> server-admins at crosswire.org
> http://www.crosswire.org/mailman/listinfo/server-admins
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/server-admins/attachments/20250814/09d93e79/attachment.htm>