Here is is a quick note on the fix I made today for a messed up Apache server.
The environment
Apache2 + Debian + mod_suexec + mod_fcgi
The problem
The server was down only for PHP-running websites. It triggered a connexion reset when trying to run PHP scripts. Static content and reverse proxies were working correctly, as if nothing was happening.
In the server logs, the following error was present :
mod_fcgid: can't lock process table
The suexec log was showing no anomaly.
The server wasn’t gracefully stopping anymore, and I had to kill him with the -9 signal to stop it.
The cause
Searching around the web was giving me very few clues on the problem. I couldn’t find why the CGI subprocess was broken, it is spawned by Apache and so it should not be missing permissions (there is no CGI daemon listening in Apache setup, contrary to Nginx)
What causes the Apache subprocesses to get spawned ? Either the Apache mpm-workers, or mpm-prefork, or mpm-event or mpm-itk. I knew I was using workers, but a glance in APT installed packages showed me broken packages on workers and ITK. Wait… ITK ?
Actually, I run Puppet, and Puppet was configured to keep Apache ot its latest version from repositories. With the release of a newer version of Apache, it had updated the apache2 package … and installed apache2-mpm-itk while apache2-mpm-workers was still present !
The result was a conflict while trying to spawn CGI processes, and a crash of the subprocess.
The fix
Doing a remove apache2-mpm-itk, and force-reinstall apache2-mpm-workers did the job.
