# Apache crashes every now and then

## baak6

Hello friends.

I'm running apache 2.0.52-r2 with SSL and PHP 4.3.10. A few days ago it started crashing for some reason, this is what the logs say:

```

[...]

[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81

[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81

[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81

[Mon Feb 21 17:09:21 2005] [alert] Child 11310 returned a Fatal error...\nApache is exiting!

[Mon Feb 21 17:09:21 2005] [emerg] (43)Identifier removed: couldn't grab the accept mutex

[Mon Feb 21 17:09:21 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:09:34 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex

[Mon Feb 21 17:11:03 2005] [emerg] (22)Invalid argument: couldn't release the accept mutex

[Mon Feb 21 17:55:18 2005] [notice] Digest: generating secret for digest authentication ...

[Mon Feb 21 17:55:18 2005] [notice] Digest: done

[Mon Feb 21 17:55:19 2005] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?

[Mon Feb 21 17:55:19 2005] [notice] Apache/2.0.52 (Gentoo/Linux) mod_ssl/2.0.52 OpenSSL/0.9.7e PHP/4.3.10 configured -- resuming normal operations

```

After re-emerging apache once it complained about not being able to use some file in /var/cache/apache2/, I checked and the apache2 dir didn't exist, so I made it and it stopped complaining and started, that's possibly related, it was something regarding SSL and that mutex thing...

Does anyone know what's going on and how to fix this? I'm going crazy over complaints about the server being down.  :Sad: 

Thanks in advance.

----------

## j-m

You need /var/cache/apache2/ for mod_ssl. There should be no problems once this missing directory exists. Check the owner and rights. 

```

drwxr-xr-x   2 apache apache      48 May 27  2004 apache2

```

----------

## baak6

It was previously

```

drwxr-xr-x   2 apache root     72 feb 21 17:55 apache2

```

I changed it to apache:apache...I hope it works, thanks!  :Smile: 

----------

## midknight_gentoo

my apache is doing the same thing now... 

did this resolve the issue???

----------

## baak6

It stopped crashing but now it started again...this i s driving me nuts.

----------

## tlarco

I also have the same thing happening.  The error sounds like a limit on the number of logins or procs a UID is allowed to run and the resource becomes unavailable.  I had some values defined in 

/etc/security/limits.conf 

that were suggested in the Gentoo Linux Security Guide (Code Listing 6.1).  I am not sure if Apache uses these vaules, but the threshold being crossed could be set somewhere in apache itself and not in Linux.  

Did you also apply hardening steps from the Gentoo Security Guide?  This is the first time I have had this problem with the 10 Gentoo servers I use every day.  The server in question is the only one with an entry in /etc/security/limits.conf.  I removed the entries and will wait to see if it happens again.  Hope this helps!

Tony

----------

## Herring42

I've been having these problems as well.

It became most noticable after I installed egroupware, especially when I connected kontact's contacts to it. It would crash every five minutes or so. Disconnecting kontact has improved things, but it still dies every so often.

----------

## midknight_gentoo

i believe now i can trace mine to after i edited the limits.conf file for teh security howto... and it must be starving apache.. so it crashes.

is there a way in that file to allow apache more processes and not the whole system.

----------

## midknight_gentoo

mine is now crashing all the damn time...

i had to disable ssl cause as soon as i go to horde it crashes right away

i removed all the stuff in the limits.conf as well....

this bothers me

----------

## geekmug

I am replying because I am in search of the same answer.. my apache is hard limited to 15 processes and nothing I do with any of the limits stuff seems to change that.

----------

## geekmug

https://bugs.gentoo.org/show_bug.cgi?id=64700

----------

## Ubnormal

Hi all!

I have the same problem, my Apache 2.0.54 craches a few moments after startup with the following lines in the error_log:

```

[Fri May 06 00:56:46 2005] [notice] Apache configured -- resuming normal operations

[Fri May 06 00:56:46 2005] [info] Server built: May  5 2005 23:51:35

[Fri May 06 00:56:46 2005] [debug] worker.c(1632): AcceptMutex: sysvsem (default: sysvsem)

[Fri May 06 00:56:46 2005] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread

[Fri May 06 00:56:46 2005] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread

[Fri May 06 00:56:47 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81

[Fri May 06 00:56:47 2005] [alert] Child 3542 returned a Fatal error...\nApache is exiting!

[Fri May 06 00:56:47 2005] [warn] child process 3521 still did not exit, sending a SIGTERM

[Fri May 06 00:56:47 2005] [warn] child process 3523 still did not exit, sending a SIGTERM

[Fri May 06 00:56:47 2005] [warn] child process 3521 still did not exit, sending a SIGTERM

[Fri May 06 00:56:47 2005] [warn] child process 3523 still did not exit, sending a SIGTERM

[Fri May 06 00:56:48 2005] [warn] child process 3521 still did not exit, sending a SIGTERM

[Fri May 06 00:56:48 2005] [warn] child process 3523 still did not exit, sending a SIGTERM

[Fri May 06 00:56:53 2005] [error] child process 3521 still did not exit, sending a SIGKILL

[Fri May 06 00:56:53 2005] [error] child process 3523 still did not exit, sending a SIGKILL
```

When i compile apache with mpm-prefork i have error_log with multiple "(22)Invalid argument: couldn't grab the accept mutex" lines.

I have  2Gb ram, 99% idle system, clear /etc/security/limits.conf, ulimit shows "unlimited". So "Resource temporarily unavailable" message in error_log looks very strange... 

It seems /etc/init.d/apache doesn't use start-stop-daemon as described in the upper post but the problem very similar. Is there any step by step instruction to solve the problem?

----------

## krimhum

I'm having the same problems and am seeking a solution.

I read on one of the apache sites that changing your AcceptMutex value will fix the issue.

I changed mine to "AcceptMutex fcntl" and am going to wait a week and see if it still crashes.

Regards,

----------

## jensb

Hi,

I had the same problem with mysql. It also used to many processes. Even though I changed the values for

```
mysql     hard      nproc      XX
```

in /etc/security/limits.conf didn't result in a change of behaviour, which I could verify in the logfiles. I use hardened sources with logging of resource violations enabled. The log still showed the same limit of YY processes that was violated To narrow down the problem I changed the wildcard entry for all users 

```
*       hard        nproc       YY
```

to a different value (ZZ) than before. Now after crashing mysql again the log showed ZZ processes as violated limit. That means that this file was the reason for the behaviour. Still I didn't understand why the system didn't see XX as limit since mysql ran as user mysql. The reason is - as mentioned earlier -

https://bugs.gentoo.org/show_bug.cgi?id=64700

When mysql is started the user is switched from root to mysql. This switch does not query the pam. I downloaded the diffs provided by the devolopers that took care of this matter and recompiled the start-stop-daemom with those changes. The start-stop-daemon starts mysql and with the new version querys the pam for the userchange. That way all the limits are set correctly.

A similar problem applies to apache, I think. The root process spawns child processes that run under the uid of the configured user (i.e. apache) but does not query the pam while doing that and therefore no limits can be set. Maybe that is also here the reason for this error.

Jens

----------

## jensb

For me the a little change in /etc/init.d/apache2 did the trick:

```

start() {

   checkconfig || return 1

   ebegin "Starting apache2"

   [ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache

   [ -f /usr/lib/apache2/build/envvars ] && . /usr/lib/apache2/build/envvars

   ulimit -u XX

   ${APACHE2} ${APACHE2_OPTS} -k start

   eend $?

}

```

I added the line: ulimit -u XX which limits the number of processes per user.

Limits don't apply for root; they are shown though when using

```
ulimit -a
```

but they have no effect. I guess these limits are now set for the session, so that all the child processes even though they're not root inherit these settings. Using this method work so far for me and now its possible to apply all the other limits like cpu-time, file-size etc. too.

Jens

----------

## Rainbow goblin

You me have rescueed, thanks!   :Very Happy: 

----------

## padde

Then - what's the solution? Will there be an update? I'm experiencing this error since today (running subversion on my apache since yesterday, so I think this has something to do with it).

http://rafb.net/paste/results/N94yxi51.html

I'm feeling _quite_ uncomfortable with a crashing webserver...

----------

## padde

It happened again today. After a subversion update (Tue Jan 29 02:20:13 2008 >>> dev-util/subversion-1.4.6). I didn't restart apache after the update (which is probably the cause), and a few hours later during a (harmless) attack consisting of around 10-20 accesses/second to some non-existing php scripts the server went down with the following log entries:

```

...

[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81 

[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81 

[Wed Jan 30 12:05:27 2008] [alert] Child 4646 returned a Fatal error... Apache is exiting! 

[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81 

[Wed Jan 30 12:05:28 2008] [emerg] (22)Invalid argument: couldn't grab the accept mutex 

[Wed Jan 30 12:05:28 2008] [emerg] (22)Invalid argument: couldn't grab the accept mutex 

...

```

Funny thing was that I immediately found this thread (and my own >1 year old post)  :Wink: 

----------

