# Linux server unstable! - out of memory?!

## oebele

I'am running an Linux mail/web/etc server.

Every night the server seems to run out of memory, and at the end just hangs/crash.

Its no hardware problem, the machine has been replaced already.

The strangest thing is that this setup/server in the past had been running ROCK solid! Never crashes or whatso ever.

I've tried to upgrade almost al of the running services to the latest build.

But things keep the same.

Here are some specs:

Kernel 2.4.19 crypto.

services running;

* samba 3

* mysql 3

* dhcpd

* proftpd

* postfix

* f-prot (daily cronned)

SYSLOG;

Jun  7 00:09:13 detonator syslogd 1.4.1: restart.

Jun  7 00:11:25 detonator postfix/smtp[11511]: 58E6C6CA85: to=<j.posthumus@kcl.znb.nl>, relay=kcl.znb.nl[213.160.196.221], delay=139, status=sent (250 OK)

Jun  7 00:19:13 detonator dhcpd: DHCPREQUEST for 192.168.2.125 from 00:10:4b:69:61:0d via eth1

Jun  7 00:19:13 detonator dhcpd: DHCPACK on 192.168.2.125 to 00:10:4b:69:61:0d via eth1

Jun  7 01:00:36 detonator postfix/pickup[11886]: 885326CA84: uid=0 from=<root>

Jun  7 01:00:36 detonator postfix/cleanup[11917]: 885326CA84: message-id=<20040606230036.885326CA84@detonator.detonator.sytes.net>

Jun  7 01:00:37 detonator postfix/qmgr[6583]: 885326CA84: from=<root@detonator.sytes.net>, size=11798, nrcpt=1 (queue active)

Jun  7 01:00:37 detonator postfix/local[11919]: warning: database /etc/mail/aliases.db is older than source file /etc/mail/aliases

Jun  7 01:00:37 detonator postfix/local[11919]: 885326CA84: to=<root@detonator.sytes.net>, relay=local, delay=1, status=sent (mailbox)

Jun  7 02:00:17 detonator postfix/pickup[11886]: 363DF6CA84: uid=0 from=<root>

Jun  7 02:00:17 detonator postfix/cleanup[11974]: 363DF6CA84: message-id=<20040607000017.363DF6CA84@detonator.detonator.sytes.net>

Jun  7 02:00:17 detonator postfix/qmgr[6583]: 363DF6CA84: from=<root@detonator.sytes.net>, size=11797, nrcpt=1 (queue active)

Jun  7 02:00:17 detonator postfix/local[11976]: warning: database /etc/mail/aliases.db is older than source file /etc/mail/aliases

Jun  7 02:00:17 detonator postfix/local[11976]: 363DF6CA84: to=<root@detonator.sytes.net>, relay=local, delay=0, status=sent (mailbox)

Jun  7 02:16:12 detonator dhcpd: Wrote 0 deleted host decls to leases file.

Jun  7 02:16:12 detonator dhcpd: Wrote 0 new dynamic host decls to leases file.

Jun  7 02:16:12 detonator dhcpd: Wrote 51 leases to leases file.

Jun  7 02:16:38 detonator proftpd[11989]: detonator (210.245.188.212[210.245.188.212]) - FTP session opened. 

Jun  7 02:16:38 detonator proftpd[11989]: detonator (210.245.188.212[210.245.188.212]) - FTP session closed. 

Jun  7 02:25:29 detonator kernel: Out of Memory: Killed process 6550 (apache).

Jun  7 02:25:37 detonator kernel: Out of Memory: Killed process 6542 (apache).

Jun  7 02:25:42 detonator kernel: Out of Memory: Killed process 6543 (apache).

Jun  7 02:25:49 detonator kernel: Out of Memory: Killed process 6544 (apache).

Jun  7 02:25:58 detonator kernel: Out of Memory: Killed process 6547 (apache).

Jun  7 02:26:15 detonator kernel: Out of Memory: Killed process 6541 (apache).

Jun  7 02:26:40 detonator kernel: Out of Memory: Killed process 6549 (apache).

Jun  7 02:26:57 detonator kernel: Out of Memory: Killed process 6545 (apache).

Jun  7 02:27:28 detonator kernel: Out of Memory: Killed process 6546 (apache).

Jun  7 02:29:17 detonator kernel: Out of Memory: Killed process 6548 (apache).

Jun  7 02:31:21 detonator kernel: Out of Memory: Killed process 11992 (apache).

Jun  7 02:32:03 detonator kernel: Out of Memory: Killed process 11993 (apache).

Jun  7 02:32:23 detonator kernel: Out of Memory: Killed process 11994 (apache).

Jun  7 02:32:40 detonator kernel: Out of Memory: Killed process 11991 (pickup).

Jun  7 02:32:50 detonator postfix/master[1520]: warning: process /usr/lib/postfix/pickup pid 11991 killed by signal 9

Jun  7 02:35:16 detonator kernel: Out of Memory: Killed process 11995 (apache).

Jun  7 02:35:28 detonator postfix/master[1520]: warning: unix_trigger_event: read timeout for service public/flush

Jun  7 02:35:41 detonator kernel: Out of Memory: Killed process 11997 (apache).

Jun  7 02:36:03 detonator kernel: Out of Memory: Killed process 11168 (named).

Jun  7 02:36:06 detonator kernel: Out of Memory: Killed process 11169 (named).

Jun  7 02:36:09 detonator kernel: Out of Memory: Killed process 11170 (named).

Jun  7 02:36:11 detonator kernel: Out of Memory: Killed process 11171 (named).

Jun  7 02:36:11 detonator kernel: Out of Memory: Killed process 11172 (named).

Jun  7 02:41:37 detonator kernel: Out of Memory: Killed process 11998 (apache).

Jun  7 02:42:02 detonator kernel: Out of Memory: Killed process 11999 (apache).

Jun  7 02:44:05 detonator kernel: Out of Memory: Killed process 12002 (apache).

Jun  7 02:44:11 detonator kernel: Out of Memory: Killed process 12003 (apache).

Jun  7 02:44:29 detonator kernel: Out of Memory: Killed process 12005 (apache).

Jun  7 02:45:18 detonator last message repeated 4 times

Jun  7 02:47:10 detonator kernel: Out of Memory: Killed process 12005 (apache).

Jun  7 02:49:59 detonator kernel: Out of Memory: Killed process 12005 (apache).

Jun  7 02:52:13 detonator kernel: Out of Memory: Killed process 12006 (apache).

Jun  7 02:55:51 detonator kernel: Out of Memory: Killed process 12006 (apache).

Jun  7 03:01:25 detonator kernel: Out of Memory: Killed process 12006 (apache).

Jun  7 02:57:22 detonator postfix/master[1520]: warning: unix_trigger_event: read timeout for service public/flush

Jun  7 03:04:28 detonator kernel: Out of Memory: Killed process 12006 (apache).

Jun  7 03:09:07 detonator kernel: Out of Memory: Killed process 12008 (apache).

Jun  7 03:08:36 detonator postfix/qmgr[6583]: fatal: watchdog timeout

Jun  7 03:13:28 detonator kernel: Out of Memory: Killed process 12008 (apache).

Jun  7 03:18:47 detonator kernel: Out of Memory: Killed process 12008 (apache).

Jun  7 03:20:12 detonator kernel: Out of Memory: Killed process 12008 (apache).

Jun  7 03:22:35 detonator kernel: Out of Memory: Killed process 12008 (apache).

Jun  7 03:23:00 detonator postfix/master[1520]: fatal: watchdog timeout

Jun  7 03:27:07 detonator kernel: Out of Memory: Killed process 12009 (apache).

Jun  7 03:29:43 detonator kernel: Out of Memory: Killed process 12009 (apache).

Jun  7 03:31:57 detonator last message repeated 2 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12011 (apache).

Jun  7 03:31:57 detonator last message repeated 3 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12012 (apache).

Jun  7 03:31:57 detonator last message repeated 7 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12013 (apache).

Jun  7 03:31:57 detonator last message repeated 3 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12014 (apache).

Jun  7 03:31:57 detonator last message repeated 3 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12015 (apache).

Jun  7 03:31:57 detonator last message repeated 5 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12016 (apache).

Jun  7 03:31:57 detonator last message repeated 2 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12017 (apache).

Jun  7 03:31:57 detonator last message repeated 3 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12018 (apache).

Jun  7 03:31:57 detonator last message repeated 4 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12019 (apache).

Jun  7 03:31:57 detonator last message repeated 2 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12020 (apache).

Jun  7 03:31:57 detonator last message repeated 6 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12021 (apache).

Jun  7 03:31:57 detonator last message repeated 4 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12022 (apache).

Jun  7 03:31:57 detonator last message repeated 4 times

Jun  7 03:31:57 detonator kernel: Out of Memory: Killed process 12023 (apache).

-----cut-------

keeps going until almost every daemon is killed en the machine is not longer responding to anything.

Probebly you won't more detailed specs.

Tell me which and i will get it after work!

Thanx for reading so far!

Hope you can help me out!

----------

## Mnemia

Sounds an awful lot like you have a memory leak in some running application. I see you're running some big server applications. How much RAM and active swap do you have on the machine?

I would keep an eye on the output of top and try to see what is consuming all the memory. There are also some other threads in the forums going into more detail about how to track down memory leaks.

----------

## oebele

 *Mnemia wrote:*   

> Sounds an awful lot like you have a memory leak in some running application. I see you're running some big server applications. How much RAM and active swap do you have on the machine?
> 
> I would keep an eye on the output of top and try to see what is consuming all the memory. There are also some other threads in the forums going into more detail about how to track down memory leaks.

 

The machine is equipped with 192mb (before 128mb, running stable over hundreds of day's uptime!)

swap 'bout 150Mb. 

I'am first getting rid of f-prot, don't trust this package ;]

----------

## smart

You might also want to check apaches accesslogs for around that time.

Maybe you are getting hit by sth. like a robot/search engine.

----------

## neuron

could try using top and sorting per memory usage, to see if you can figure it out.

if not you could try to remove the memory you added, could be it's bad.  Though you would probably get other errors.

----------

## jkt

as the server was "rock solid", the problem might be caused by upgrade of some daemon. are you running ~arch?

----------

## oebele

 *jkt wrote:*   

> as the server was "rock solid", the problem might be caused by upgrade of some daemon. are you running ~arch?

 

Yes  :Confused:  i've got that flag on...

But the're have been no daemons updated.

The system is heavily loaded with crons, and i'have seen

that the memory problems start after running these crons

So i removed most of them and now just see if it survives the night!

I'll keep you informed.

----------

## Janne Pikkarainen

I suspect that some php/perl/whatever script is being a nasty boy, eating up all the memory it can get. I've seen that happening many, many times and if setting up memory limits (for example) in php.ini doesn't help, you may want to use pam_limits and/or standard ulimit function in shell, so those processes would be killed before they kill your server.

----------

## jkt

and PHP needs to be compiled with PHP support, of course, ie `USE=memlimit emerge php`. you should try to run these crons at different times and increase the swap amount.

----------

## oebele

Well, the server is still alive after one night. 

Thanks for the tips! It could well be a perl script yes.

I'don't now how cron handles multiple task, for example i've got a couple of daily task, are they being run seq. or parallel?

This was the cron list

webalizer

emerge sync

slocate 

some DHCP-statistics

f-prot full scan

If the server keeps stable without these crons being run i'll put them back one by one and keep a eye on 'top' of the processes =:-]

----------

## Mnemia

 *oebele wrote:*   

> Well, the server is still alive after one night. 
> 
> Thanks for the tips! It could well be a perl script yes.
> 
> I'don't now how cron handles multiple task, for example i've got a couple of daily task, are they being run seq. or parallel?
> ...

 

I don't know for sure about the others, but it's not a very good idea to put emerge sync in a cron job. Not only do you put unnecesary strain on the Gentoo mirrors, but you also waste a lot of resources on your server. Syncs can take a long time to complete if you happen to hit a rather slow mirror in the rotation.

Better to just do a sync when it's necessary: when you actually install or update packages on your machine. That way you also have a tree that is more consistent with the level of "updatedness" that your system has as far as installed packages. If you insist on putting this in a cron job, at least make it weekly instead of daily.

----------

## slestak

 *Quote:*   

> 
> 
> keeps going until almost every daemon is killed en the machine is not longer responding to anything.
> 
> Probebly you won't more detailed specs.
> ...

 

Do you have squirrelmail installed?  There is a nasty bug (thats been fixed in 1.4.3a) in the reply-to php script that consumes all memory on the webserver.

----------

## oebele

Nope, no squirrelmail.

The server is still running   :Cool:  after 2 days.

I'am letting it run for a while now and put in the crons later.

It's my guess that webalizer or f-prot is a little uncarefull with the 'huge' amounts of memory onboard =>;]

----------

## oebele

After 14 day's the machine is still running stable.

I've checked the memory usage and came to the conclusion that f-prot is the cron job thats caused the memory leak.

Thanx for all your help!

----------

