# HELP - cannot allocate memory; had to reboot

## Moriah

I had a machine die in a weird way this afternoon.  This machine has been up for weeks if not months.  All it runs is 2 instances of tcpdump, and gets logged into via ssh every 5 minutes to fetch disk drive temperatures.  Apparently, something went wrong when PAM tried to authenticate the ssh user, because PAM could not open the pam library to hook into the pam functions.  Here is what was in /var/log/messages:

```

Jul  5 11:53:30 ezekiel sshd[21549]: SSH: Server;Ltype: Version;Remote: 192.168.1.88-56244;Protocol: 2.0;Client: OpenSSH_5.9p1-hpn13v11

Jul  5 11:53:30 ezekiel sshd[21549]: SSH: Server;Ltype: Kex;Remote: 192.168.1.88-56244;Enc: aes128-ctr;MAC: hmac-md5;Comp: none [preauth]

Jul  5 11:53:30 ezekiel sshd[21549]: SSH: Server;Ltype: Authname;Remote: 192.168.1.88-56244;Name: root [preauth]

Jul  5 11:53:30 ezekiel sshd[21549]: Accepted publickey for root from 192.168.1.88 port 56244 ssh2

Jul  5 11:53:30 ezekiel sshd[21549]: pam_unix(sshd:session): session opened for user root by (uid=0)

Jul  5 11:53:31 ezekiel sshd[21549]: Received disconnect from 192.168.1.88: 11: disconnected by user

Jul  5 11:53:31 ezekiel sshd[21549]: pam_unix(sshd:session): session closed for user root

Jul  5 11:59:01 ezekiel sshd[21557]: SSH: Server;Ltype: Version;Remote: 192.168.1.88-58138;Protocol: 2.0;Client: OpenSSH_5.9p1-hpn13v11

Jul  5 11:59:01 ezekiel sshd[21557]: SSH: Server;Ltype: Kex;Remote: 192.168.1.88-58138;Enc: aes128-ctr;MAC: hmac-md5;Comp: none [preauth]

Jul  5 11:59:01 ezekiel sshd[21557]: SSH: Server;Ltype: Authname;Remote: 192.168.1.88-58138;Name: root [preauth]

Jul  5 11:59:02 ezekiel cron[21561]: PAM unable to dlopen(/lib64/security/pam_unix.so): /lib64/security/pam_unix.so: cannot map zero-fill pages: Cannot allocate memory

Jul  5 11:59:02 ezekiel cron[21561]: PAM adding faulty module: /lib64/security/pam_unix.so

Jul  5 11:59:02 ezekiel cron[21561]: PAM unable to dlopen(/lib64/security/pam_limits.so): /lib64/security/pam_limits.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel cron[21561]: PAM adding faulty module: /lib64/security/pam_limits.so

Jul  5 11:59:02 ezekiel cron[21561]: PAM unable to dlopen(/lib64/security/pam_deny.so): /lib64/security/pam_deny.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel cron[21561]: PAM adding faulty module: /lib64/security/pam_deny.so

Jul  5 11:59:02 ezekiel cron[21561]: Module is unknown

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_access.so): /lib64/security/pam_access.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_access.so

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_cracklib.so): /lib64/security/pam_cracklib.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_cracklib.so

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_loginuid.so): /lib64/security/pam_loginuid.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_loginuid.so

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_lastlog.so): /lib64/security/pam_lastlog.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_lastlog.so

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_limits.so): /lib64/security/pam_limits.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_limits.so

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_motd.so): /lib64/security/pam_motd.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_motd.so

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_mail.so): /lib64/security/pam_mail.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_mail.so

Jul  5 11:59:02 ezekiel sshd[21557]: PAM unable to dlopen(/lib64/security/pam_deny.so): /lib64/security/pam_deny.so: failed to map segment from shared object: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21557]: PAM adding faulty module: /lib64/security/pam_deny.so

Jul  5 11:59:02 ezekiel sshd[21557]: fatal: Access denied for user root by PAM account configuration [preauth]

Jul  5 11:59:02 ezekiel sshd[21562]: SSH: Server;Ltype: Version;Remote: 192.168.1.88-58139;Protocol: 2.0;Client: OpenSSH_5.9p1-hpn13v11

Jul  5 11:59:02 ezekiel sshd[21562]: SSH: Server;Ltype: Kex;Remote: 192.168.1.88-58139;Enc: aes128-ctr;MAC: hmac-md5;Comp: none [preauth]

Jul  5 11:59:02 ezekiel sshd[21562]: SSH: Server;Ltype: Authname;Remote: 192.168.1.88-58139;Name: root [preauth]

Jul  5 11:59:02 ezekiel sshd[21562]: Accepted publickey for root from 192.168.1.88 port 58139 ssh2

Jul  5 11:59:02 ezekiel sshd[21562]: pam_unix(sshd:session): session opened for user root by (uid=0)

Jul  5 11:59:02 ezekiel sshd[21562]: Received disconnect from 192.168.1.88: 11: disconnected by user

Jul  5 11:59:02 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 11:59:02 ezekiel sshd[21562]: pam_unix(sshd:session): session closed for user root

Jul  5 12:00:01 ezekiel cron[21569]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons)

Jul  5 12:00:01 ezekiel sSMTP[21571]: Could not find password entry for UID 0

Jul  5 12:00:01 ezekiel cron[21568]: (root) MAIL (mailed 50 bytes of output but got status 0x0001

Jul  5 12:00:01 ezekiel )

Jul  5 12:04:32 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:04:32 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:04:32 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:10:01 ezekiel cron[1254]: (CRON) error (can't fork)

Jul  5 12:10:06 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:10:06 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:10:06 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:15:39 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:15:39 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:15:39 ezekiel sshd[1223]: error: fork: Cannot allocate memory

Jul  5 12:20:01 ezekiel cron[1254]: (CRON) error (can't fork)

```

Does anybody know what happened here?    :Question: 

I had to reboot to get the box going again.    :Evil or Very Mad: 

----------

## Shum

Sounds like you ran out of RAM.

Probably, something you're running is slowly leaking memory which means this is going to happen periodically every few weeks/months. Next time you've left box up and running for a few weeks run top and see if there is a process that is using up an unusually large amount of memory. That'll tell you who the culprit is. Then upgrade that package to see if it fixes it and if it doesn't then file a bug report against it.

----------

## Moriah

No, top did not show anything unusual at the time of the crash, so I rebooted.  This morning, it was dead again, but with a panic.  I am now running memtest86 on it.    :Sad: 

----------

