[Server-admins] Server has huge java task at 100% CPU, and vmstat 87% wa

[Jonathan Marsden]

The crosswire.org server has some puzzling load characteristics:

(a) top shows a java process (Tomcat) using 100% CPU, all the time.
This process has a virtual memory size of over 8GB, and a resident set
size of over 1GB.  What is it *doing* with all that CPU and RAM?

(b) Possibly related (?), running the standard:

  vmstat 5

shows the server is waiting for I/O 87% of the time.  That's a *lot* of
I/O waiting.  Again, what is causing this?  I'd expect to see large
amounts of idle time, not io wait time.

Instinctively, this looks like something within Tomcat is misconfigured
and "running away" doing something it need not really do, but the server
is so vastly oversized for the task at hand that we hardly notice the
issue -- there are 7 other cores doing nothing, and we aren't doing
anything else all that I/O intensive on this machine!  Even though there
are apparently no obvious bad effects from the 100% CPU / 87% wait, they
offend my sense of what "should" be observed on a server like this one.

Any ideas what is causing this, or how to investigate it further?

Jonathan