#StackBounty: #nginx #mysql #php-fpm #mariadb MariaDB opens lots of files causing PHP-FPM to fail

Bounty: 50

There is a problem I’m dealing with that under a known traffic (e.g. reaching 300 concurrent users) server doesn’t handle more incoming requests due to open files. This is the current limit:

$ cat /etc/security/limits.conf
* - nofile 1000000

Error comes into sight from within php-fpm error.log file:

[16-Aug-2019 02:14:12] WARNING: [pool ****] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 99 idle, and 506 total children
[16-Aug-2019 02:14:12] ERROR: failed to prepare the stderr pipe: Too many open files (24)

At this moment nginx responds with 499 status code.

I can see that when no requests are made mysqld has this number of open files (default number?):

$ lsof | grep '^mysqld' | wc -l
104104

While under load it goes up to a number like 603035 and then those errors from php-fpm trigger.

Server doesn’t reach 10% CPU load and most of RAM is free. How can I avoid such number of open files or if not possible what should be done and investigated in this situation?


Get this bounty!!!

#StackBounty: #nginx #php #php-fpm #wordpress #amazon-lightsail PHP-FPM + nginx producing a permission denied, but only on certain larg…

Bounty: 50

I have an AWS Lightsail instance (1GB RAM instance) running a relatively new website (i.e. virtually no traffic). It’s running nginx and PHP-FPM 7.3 (tried with 7.2 as well) and MariaDB. All of this is under CentOS 7.

Everything worked fine under the AWS free tier. I ran a T2.micro EC2 instance and a T2.micro RDS instance. Lightsail has been a bit… touchier. To make Lightsail work, I switched PHP-FPM to ondemand

ondemand – no children are created at startup. Children will be forked when
new requests will connect.

I had to do this, or MariaDB would randomly crash. This doesn’t appear to affect the problem below.

The WordPress admin panel stopped working properly and everyone said to turn CONCATENATE_SCRIPTS off. That works… mostly. The editor for both posts and templates malfunctions. Nobody has been able to give me a clue why. In looking around, I found something myself.

The pages not working are not loading completely. With CONCATENATE_SCRIPTS on, CSS files are loaded in one giant page. Because this fails to completely render, the CSS and JS files are ignored by the browser. CONCATENATE_SCRIPTS works around that by simply splitting them into component files, which are smaller and load easily. But the edit page cannot be split, and debugging the underlying issue has been maddening. I get a 200 response and some data

Page returns 200

But the page draw is incomplete. I’d say maybe 80-90% of the HTML is there, but cut off. In the section starting here (JS block)

wp.apiFetch.use( wp.apiFetch.createPreloadingMiddleware( {"/":{"body":{"name":"S

it just abruptly ends, and at the same point every time. It’s as if PHP-FPM or nginx has just stopped, but without any error logs (and most of the other issues out there about this type of setup are for pages not drawing at all). Even stranger is it’s not doing it to smaller pages, just really long ones. There’s no steal time in top and the instance doesn’t seem to be under any serious load, so I’m not sure why it would do that. I reloaded all files fresh, and even set up a separate WP site to test this and they all do it.

Per comments, I turned on nginx debug logging and found

2019/08/07 02:33:08 [crit] 1461#0: *47 open() "/var/lib/nginx/tmp/fastcgi/3/00/0000000003" failed (13: Permission denied) while reading upstream, client: x.x.x.x, server: example.com, request: "GET /wp-admin/post-new.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000",

This doesn’t make any sense why it would do this on JUST a handful of large files. No drive is full or close to it. I read this question but both nginx and PHP-FPM are running under apache. Deleting the tmp files didn’t fix it either. The directories are owned by apache:root, but changing them to apache:apache has no effect. SELinux doesn’t seem to be the culprit either. I’m also not using proxy_cache.


Get this bounty!!!