#StackBounty: #ubuntu #php #apache-2.4 #apache2 Diagnosing apache cpu / memory / process spike

Bounty: 50

I have a server running Ubuntu 18.04 which has been experiencing huge CPU spike that nearly brings Apache to a halt once or twice a day. The server runs a couple of websites – all php & mysql driven applications. Here’s a bit more detail on things I’ve looked into:

MySQL: Slow query log is enabled and set to log queries taking longer than 1 second. Reviewing this log after a spike reveals nothing in particular. No long running queries to speak of.

CRON: I’ve reviewed all user cron jobs running on the server and there’s nothing that happens during the times when these spikes occur. There are only a couple of CPU intensive jobs and they run around 3am and take approx 5 minutes to complete.

PHP: Both max_input_time and max_execution_time are set to 60 seconds and memory_limit is 64M (this is a 16GB server which typically doesn’t come close to maxing out memory usage).

APACHE: Our host (Linode) has a tool called Longview which shows various diagnostics related to Apache. Despite the huge spike in resources consumed, requests seem to be happening at a normal rate. Manually inspecting the access logs confirms this. Here’s a screenshot of the Apache tab in Longview showing a spike in Workers, CPU and RAM this morning – as well as a relatively normal rate of Requests:

enter image description here

I’ve also added flags in the Apache access logs to show time and I/O data about each request. The end of the LogFormat is time:%T input:%I output:%O. None of the request or response sizes are unusually large (1MB might be the largest response I saw and that was for an image). The only thing standing out is the "time taken to serve the request" which is the %T flag. At a certain point in the morning many seemingly normal requests take 5 – 10 minutes to complete for no apparent reason.

I’m completely stumped at this point. Where can i go from here to diagnose the event that’s triggering this?

Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.