# I think my SMTP server got compromised(situation escalated)

## Bigun

I'm not 100% certain of this.  I have been running this server for years (I do a systemwide update about once a year), and I would see a few attempts for people trying to use my SMTP server to send mail, but they always failed.

This morning I get up and check my mail and find about 4,000 items in my inbox and ALL of them are various undelivered mail messages.

I ssh'd into the system and stopped postfix.

From here I'm doing some investigation and could use some help.

From what I can tell, the user found this user to use somehow: "apache@mail.<mydomain>.com"

I was not aware of that user even existing in my v-mail system.  I just double checked and I did not see apache as a real or virtual mailbox.

Help?

----------

## py-ro

Unsecure php-script or cgi?

Py

----------

## Anarcho

If PHP is sending emails, it normally does this with a sender name of apache@<hostname> on gentoo (on other linux boxes the user running apache would be named http or www-run). So this is most probably an insecure PHP script. Check the log files.

----------

## Bigun

Now I'm getting worried.  I see this in my log after I shut off postfix:

```
Aug 26 07:12:55 pwnedclips su[928]: pam_unix(su:auth): authentication failure; logname= uid=81 euid=0 tty=pts/1 ruser=apache rhost=  user=root

Aug 26 07:12:57 pwnedclips su[928]: FAILED su for root by apache

Aug 26 07:12:57 pwnedclips su[928]: - pts/1 apache:root

```

Did this person get ssh access through a user called apache?

----------

## Bigun

I can see where he was trying to get in... and seeing as how he's attempting to break in, I'm not going to conceal the IP:

```
Aug 26 10:21:20 pwnedclips sshd[7286]: pam_tally(sshd:auth): Tally overflowed for user root

Aug 26 10:21:20 pwnedclips sshd[7286]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=190.210.45.181  user=root

Aug 26 10:21:22 pwnedclips sshd[7286]: Failed password for invalid user root from 190.210.45.181 port 33677 ssh2

Aug 26 10:21:23 pwnedclips sshd[7288]: reverse mapping checking getaddrinfo for customer-static-210-45-181.iplannetworks.net [190.210.45.181] failed - POSSIBLE BREAK-IN ATTEMPT!

Aug 26 10:21:23 pwnedclips sshd[7288]: User root from 190.210.45.181 not allowed because not listed in AllowUsers

```

----------

## ianw1974

You can install denyhosts, that way any failed attempts will block him from trying again.

I always also disable root access within the /etc/ssh/sshd_config file and I also disable the ability to use password authentication and just use SSH keys for authorised access.  No key, no access.

----------

## Bigun

 *ianw1974 wrote:*   

> You can install denyhosts, that way any failed attempts will block him from trying again.
> 
> I always also disable root access within the /etc/ssh/sshd_config file and I also disable the ability to use password authentication and just use SSH keys for authorised access.  No key, no access.

 

I only allow one user access via ssh, and it's not root.

I'm still scratching my head how he's running 'su'

----------

## ianw1974

Enable the su logging in /etc/login.defs if nothing is showing in relation to successful or failed su attempts.

My log files show this:

```
Aug 26 19:34:59 elise su[8860]: pam_unix(su:auth): authentication failure; logname=ian uid=1000 euid=0 tty=/dev/pts/0 ruser=ian rhost=  user=root

Aug 26 19:35:01 elise su[8860]: pam_authenticate: Authentication failure

Aug 26 19:35:01 elise su[8860]: FAILED su for root by ian

Aug 26 19:35:01 elise su[8860]: - /dev/pts/0 ian:root

Aug 26 19:35:03 elise su[8862]: Successful su for root by ian

Aug 26 19:35:03 elise su[8862]: + /dev/pts/0 ian:root

Aug 26 19:35:03 elise su[8862]: pam_unix(su:session): session opened for user root by ian(uid=1000)
```

EDIT:

Sorry noticed you had this already and he's coming in via user apache.

----------

## Bigun

I can see where he is trying to run it over and over again.

I have "Register Globals" off, and I can't think of any code he could have used to get in.  Even looking at the raw log, I see no entry given, it just immediatly shows attempts using 'su' out of nowhere:

```
Aug 26 07:12:55 pwnedclips su[928]: pam_unix(su:auth): authentication failure; logname= uid=81 euid=0 tty=pts/1 ruser=apache rhost=  user=root

Aug 26 07:12:57 pwnedclips su[928]: pam_authenticate: Permission denied

Aug 26 07:12:57 pwnedclips su[928]: FAILED su for root by apache

Aug 26 07:12:57 pwnedclips su[928]: - pts/1 apache:root

Aug 26 07:12:57 pwnedclips su[929]: pam_unix(su:auth): authentication failure; logname= uid=81 euid=0 tty=pts/1 ruser=apache rhost=  user=root

Aug 26 07:12:59 pwnedclips su[929]: pam_authenticate: Permission denied

Aug 26 07:12:59 pwnedclips su[929]: FAILED su for root by apache

Aug 26 07:12:59 pwnedclips su[929]: - pts/1 apache:root

```

I've now shut off apache.  I'm co-hosting a few sites that I don't see as being too important anymore that I'll stop hosting, which should narrow the site list down to 1 site.

I still want to know how he got in.. I don't anything with the apache log that looks suspicious.

----------

## Bigun

I think I found it.

It was an old expired site I was hosting, after the domain expired I removed the site's vhost entry from the apache config.

The site ran an old version of PHP-BB that hasn't been updated in a LONG time.

After sifting through the logs, I found where the connection had been made via 127.0.0.1 to authdaemond.  He had gotten the password from the old site files and found that I had left the entries in the virtual mail system for the old board.  The old e-mail entries have now been removed. 

A few questions:

How did he get access to a site that no longer had vhost entries?  Does this mean old site data needs to be removed after the domain expires?

If he got SMTP access, is there any other damage he could have done?  After he found that one entry, I found these commands being ran right after he got in, I'm assuming it's a script:

```
Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: /usr/lib/postfix

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  /usr/lib/postfix

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: daemon_directory = /usr/lib/postfix

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: command_directory = (notfound)

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: /usr/sbin

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  /usr/sbin

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: command_directory = /usr/sbin

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: queue_directory = (notfound)

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: /var/spool/postfix

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  /var/spool/postfix

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: queue_directory = /var/spool/postfix

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: process_id_directory = (notfound)

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: pid

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  pid

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: process_id_directory = pid

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: inet_interfaces = all

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: all

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  all

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: proxy_interfaces = (notfound)

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: 

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: proxy_interfaces = 

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: double_bounce_sender = (notfound)

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: double-bounce

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  double-bounce

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: double_bounce_sender = double-bounce

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: default_privs = (notfound)

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: nobody

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  nobody

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: default_privs = nobody

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: alias_database = (notfound)

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: mac_parse: hash:/etc/mail/aliases

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_eval: const  hash:/etc/mail/aliases

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_update: alias_database = hash:/etc/mail/aliases

Aug 26 01:19:05 pwnedclips postfix/smtp[9730]: dict_lookup: mail_release_date = (notfound)

```

----------

## Bigun

Also just found this in the apache access log:

```
SEARCH /\x90\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\

xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\

xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\

xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\

xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\
```

Over and over again for several pages.

----------

## Bigun

I wanna choke this guy.  To top it all off, he has a BUNCH of mail queued up and I can't start postfix without spamming everyone and their mother.

How do I clear the postfix queue?

edit:

I got it cleared:

```
postsuper -d ALL
```

----------

## ianw1974

One other thing, maybe install rkhunter and do a check in case any root kit got dropped.

With apache I also use modsecurity to secure against access.  A server can usually be access via IP, but you can stop this with modsecurity so that the domain name has to be used.  I only have one single static IP, so the configuration of ServerName and ServerAlias in Apache helps me with the vhost redirecting to the correct site.  Perhaps he gained access to the old site via the IP address.  So maybe give modsecurity a go also to help protect your web server against attacks.

----------

## Bigun

Thanks for the tip, I ran the tool and got a few warnings, but nothing was detected:

```
grep "Warning" /var/log/rkhunter.log

[08:19:05] Warning: Checking for prerequisites               [ Warning ]

[08:19:06] Warning: WARNING! It is the users responsibility to ensure that when the '--propupd' option

[08:20:05] /usr/bin/ldd                                      [ Warning ]

[08:20:06] Warning: The command '/usr/bin/ldd' has been replaced by a script: /usr/bin/ldd: Bourne-Again shell script text executable

[08:20:36] /usr/bin/whatis                                   [ Warning ]

[08:20:37] Warning: The command '/usr/bin/whatis' has been replaced by a script: /usr/bin/whatis: POSIX shell script text executable

[08:20:41] /usr/bin/lwp-request                              [ Warning ]

[08:20:41] Warning: The command '/usr/bin/lwp-request' has been replaced by a script: /usr/bin/lwp-request: a /usr/bin/perl -w script text executable

[08:44:18]   Checking loaded kernel modules                  [ Warning ]

[08:44:18] Warning: No output found from the lsmod command or the /proc/modules file:

[08:45:43]   Checking if SSH root access is allowed          [ Warning ]

[08:45:43] Warning: The SSH configuration option 'PermitRootLogin' has not been set.

[08:45:54]   Checking for hidden files and directories       [ Warning ]

[08:45:55] Warning: Hidden directory found: /dev/.udev

[08:45:55] Warning: Hidden file found: /usr/share/man/man5/.k5login.5.bz2: bzip2 compressed data, block size = 900k

```

The "PermitRootLogin" is a moot point on my system seeing as how I'm using "AllowUsers" and root is not in that list.

As for the rest, I did a test by doing an emerge --oneshot on two of those files that produced warnings, and nothing changes.  Do you get those same warnings?

----------

## Bigun

I thought things looked clean.  I did a double check by running rkhunter again and everything looked clean.

I get off work and check my mail and I was unable to get on my IMAP server.  I attempt to SSH in, no response.  No Ping, nothing.

I called the co-location guys and they stated that my server broke a 10 Mbit choke set on the server and was consuming 35Mbit of bandwidth and crashed every client he had and put two of their switches to the ground.  When they discovered it was my machine they took it offline.  I apologized and I really don't blame them for it.

I will head down there tomorrow with a laptop and ssh into the machine and see what is going on.

Any advice on what to look for?

On a side note, it's obvious my machine was compromised somehow, I will be re-installing the OS.  But I just want to know what happened so I can avoid it happening again

----------

## Hu

If you have the storage capacity, grab a drive image before you wipe the system, so that you can continue forensics at your leisure.

----------

## ianw1974

Things I would suggest once you've got the machine reinstalled and up and running.

1. Install modsecurity to protect apache from any usual attempted attacks.

2. Install rkhunter and scan your machine.  You can run the --propupd once you are sure all is OK, after a clean install this will be fine.

3. Ensure you disable root access with the relevant option in sshd_config.

4. Use keys to authenticate with SSH, so that password auth cannot be obtained.  The only downside with this is if you erase your home machine that has a key, and forgot to back it up, you can't SSH into your box again until you generate a new key and have it installed on the server to allow you access.  I've got mine safely backed up so I'll not have this problem as I did do it once  :Smile:  pick a user other than root for this key to be used for access, for example your username, or admin, or whatever.

5. Run SSH on a different port, perhaps a higher port would be better than the standard port.  Eg: 10022, but it can be anything you want.

Generally with this you should be fine.  This is what I've been using on my server for the last few years.  And as already said get a copy of it so that you can check it out at your own leisure at home on a spare machine and find out what and how they did it.

One other thing I did, since I have my own physical server, is that I have a Xen Server running, and four virtual servers.  That way, if I lose the virtual server, I can commission another new server remotely without a visit to the site.  Also, as I have all my stuff segregated, I won't lose everything in one go.  For example, I have a Backup Server, Mail Server, Proxy Server and Web Server.  Therefore, if someone now hacks and destroys my web server, I don't lose the rest of my servers.  Before I had everything in one place, so was more vulnerable as I'd lose the whole lot in one go.  I like virtualisation.  My Xen Server doesn't have any remote access other than SSH so it is impossible to connect to it the way I have it configured as mentioned here.

Also, once up and running, use nessus to scan it so you can see if any vulnerabilities are showing on any of the listening ports.

----------

## Bigun

Not sure if I need to post another post, but I figure I would ask here since I am doing this as a direct result of all the action on this thread.

Anyone know how to backup/restore a database from raw files?

----------

