# My gentoo box may have been hacked

## weyhan

Hi,

Im seeking security expert for some advice.

I have a Gentoo box at home that is my router to the Internet. It connects to the internal network as well as my DMZ. It is also the time server and firewall as well as manage my ADSL modem.

Ive just notice a gap of 6 hours in my /var/log/message recently. Right after the missing log entries, the log shows it rebooted. Im 100% sure I did not reboot it during that time.

I did try rkhunter but turn up nothing.

What should be my level of alarm right now? How else can I try to figure out what happen during that 6 hours of missing logs?

Thanks.

----------

## nixnut

Did you update baselayout on that machine recently? Or the kernel? Perhaps something changed wrt timezone or hardware clock setting. If you think somebody compromised your machine the only way to clean it is to wipe the disk and install from a known good install media. Any other measures may find some issues, but you can never be sure your machine will be free from unwanted infestations.

----------

## weyhan

nixnut,

Nope. I have not updated the router in over a year (which is why it may have been hacked...   :Embarassed: ).

I do not live in a place that observes DST. I use ntpd so it does not seem possible to have very drastic time changes. 

Besides, that also does not account for the reboot right after the missing log entries.

----------

## xtlosx

look around at all of the shells and user binaries, see if they are not linking to anything weird, with an ls -al.. just take some time, unplug the system and do some forensics... get intimate with the operating system.... dig, and dig and dig...  :Smile:   compare it to a system you know is not compromised... but yes, reinstall after you find your problem, and plug your security hole!

----------

## NeddySeagoon

weyhan,

look in /home/<username>/.bash_history for all values of <username> and /root/.bash_history

If any of them are very short be very suspsious unless you know why. They should contain about the last 300 commands executed by the user.

If your hacker was careful to edit log files, they will have been edited or removed .bash_history too.

You should be doing all your forensics on a read only disk image, so normal operation does not overwrite anything.

It will take you several days to find out how your hacker got in and you will need to recover deleted file fragments for .bash_history to discover the commands they executed.

----------

## weyhan

xtlosx,

I did snoop around but have yet to find anything suspicious apart from the missing log entries. As for plugging the hole, I would not know how to patch a hole that I do not know.  :Smile: 

NeddySeagoon,

The .bash_history was the first place Ive look and its clean. I too suspect I wont find anything in the .bash_history because the history file is the easiest to clean up. All the intruder has to do is to kill the shell with -9 to avoid saving the history to the file. If that is the case, there may not even be any fragments on disk. Besides forensics is never my strong point. I was hoping I would find any suspicious beyond the missing log entries but none so far.

Also, I find it strange that the intruder did a reboot, if there really is an intruder. I think they will usually try to preserve the uptime unless some mishaps happen and my box just rebooted. Other wise, that would be a dead give away that something happened to the box.

Arrr all those question but no answer.

----------

## cynric

I'd err on the side of caution and assume that you've been compromised and start working on a remedy to fix the current damage (after you find it) and implement preventative features. Aside from saying what others have already mentioned about analysis, I just wanted to stress the importance of doing analysis of the drive in a read-only state. Whether you make an image of it (probably overkill for most people) or just booting from a livecd and mounting the partitions as read-only, it really is worth it. There are a lot of tools out there that can help with analysis depending on how much time you want to devote to the process.

However, if it's a system that hasn't been updated in a year, one could take a reasonable guess that once you reinstall a fresh (and updated) copy of the OS the security flaw that you had will no longer be there (of course others will take its place). So, for the sake of time, you could probably just re-install. I'd suggest trying out a hardened install if you haven't already along with some additional logging procedures; either set up some form of Intrusion Detection System, remote logging, email alerts, etc. Running something like aide or keeping a md5 checksum of your freshly installed system so that you can check what files have been modified will also help give you an idea of what happened even when logs fail.

Aside from rkhunter, chkrootkit is also in the portage tree and does similiar. Dunno how much the two tools overlap with regards to their scanning method, but it's usually a good idea to use multiple tools. Good luck on the resolving the issue; it is a very time consuming task unless you have tools and experience.

----------

## weyhan

cynric,

Thanks for the advice. Ill make sure Ill try setting up some IDS as you have suggested. Ive been wanting to do that but procrastination was the key factor holding me back.

As for now, Im trying to get a firewall/router distro that I can run from CD so that I can do the proper forensics on the HD.

Thanks guys. Ill try to update my experience later.

----------

## didl

This may help you.

----------

## xtlosx

Ya, that or you can look into backtrack STD, has some good forensics applications... good luck, let us know how it turns out.... were you running any other services on this machine in question? apache, mysql, anything?

----------

## Akkara

Could there have been a power outage to your server?  You say 'at home' and I'd guess you'd have noticed a blackout but is ther any chance that that circuit might have gone out?  Just guessing at other things that might explain missing logs followed by a reboot.

----------

## weyhan

didl,

Ive found quite a few including Helix. Im guessing Helix is highly recommended.  :Very Happy: 

xtlosx,

Okay, I did find backtrack STD too.

The box in question is a dedicated router/firewall. All the services I have on this box are related to serving the Internet, routing, etc.

rp-pppoe  to dial my ADSL modem

shorewall  my firewall

ntp  to make sure the time on this box is correct and to sync the time on other boxes internally only

dhcpd  to serves ip address for my internal network

djbdns  to cache external dns queries as well as serving internally some authoritative dns entries for a few internal boxes

nut  for UPS monitoring and shutdown on power failure.

Those are besides the usual services like cron, syslog, etc.

Akkara,

No way it can be a power problem.

1. I was at home during the period where the logs were missing and there were no black out.

2. The box is on UPS and monitored by nut. There should be a log entry from nut unless I have misconfigured nut (not impossible though).

3. The bios setting for power outage is to stay down so it will not have rebooted without me pressing the power button or issued a reboot on the box.

4. I have not installed or configured anything on the box to automatically reboot without my intervention.

So no, power issue is not at play here.

Thanks for the suggestion though. Power issue is always a possibility that have to be ruled out.

----------

## NeddySeagoon

weyhan,

Hmm UPS.  

Thats a double edged sword. Failing batteries can cause odd reboots if you have the sort of UPS that 'floats' the backup supply.

It would not explain 6 hours of missing logs though.

When was the last time you rehearsed your UPS ?

----------

## Cyker

Depends on your paranoia level.

Sadly, most crackers don't know their ass from their elbow - This is the script-kiddie group.

If they got into your system, there will be really obvious evidence everywhere - The .bash_history, weird new user accounts, inconsistent log entries (esp. if, like me, you have lots of logs generated and some are sent to remote systems).

At the other end of the spectrum, if you got hacked by someone totally elite and methodical, there's no way in hell you'll be able to tell because they'd have cleaned their tracks and root-kit'd your system.

I've had the occasional paranoia-trigger with my current system (I am still not 100% confident in TCPWrappers and IPTABLES abilities...), but only once have I found any actual evidence of an intrusion, and that one was so amateurish I nearly topped myself for leaving such a stupid hole.

I monitor all my out-going traffic through a hardware router, and haven't noticed any suspicious address connects/incomings, so for the moment I still feel fairly confident that the box is uncompromised. Or if it is, it isn't being used for anything bad  :Wink: 

If you're quite paranoid, then the only real recourse is to rebuild the system of Known clean install discs, but this gets old really quickly when you're at that level of paranoia and don't trust your attempts at securing the system...

----------

## xtlosx

I agree with Cyker... It doesn't sound like you have definitive proof that your box was hacked.... I mean ya there is some weird log errors, but hey, stranger stuff has been known to happen..  Would be a shame to have to go and reinstall the box and everything on it for NO reason right?

Make sure it was an intrusion, since it's your fw\router you might want to look into Snort, and even Snort Inline... Depending on what you are trying to do this may or may not be useful to to or above your head... But Snort keeps me informed as to what is out there on my external interface as well as what it going out my internal interface, so you would see if people are doing wild spam send-outs through you and what not.. Plus, if you see something specific, run tcpdump on the box, see if you notice anything, write your own rule...

The likelihood of you getting hacked by someone very intelligent and with a purpose is low, I mean let's face it, who the hell wants my box or yours besides script kiddies who want to spam.. let's be honest here.. Good luck! Let us know what happens eh?

----------

## weyhan

NeddySeagoon,

Yes, that was my line of thinking. Err Im afraid I dont even know what is rehearsing.

Anyway, I have faith in my UPS because I just had a black out which happened to be the day after the missing logs Yes, Im sure the black out happened one day after the reboot and missing log entries because there is one day worth of logs after the reboot when the power came back. The router was on UPS for about 3 hours mostly idle and no reboot.

That, incidentally, was how Ive found the hole because I was checking to make sure the system was running fine.

Cyker,

That is also what Ive thought. On one hand, if my box is hacked, there is evidence of cleanup but a sloppy cleanup because it left a gaping hole in the logs. If it was a script kiddy, I would expect a lot more trash lying around.

You know, its my own fault that I have not made an md5 check sum image of the system

xtlosx,

I never said for sure my box was hacked but the fact that I see something suspicious and feel that there is a possibility that the box may have been hacked.

I also feel that if not for sending spam or taking control of the system to become a botnet for some DDoS attack, I cant think of why anyone would want to hack my router.

However, I have not told the whole story because I am still investigating. Ill give more details on my network setup in the next post because it is kind of long I think.

----------

## weyhan

Okay, Ill give some more details about how my home network is setup and maybe security experts here can also help point out if the setup is flawed.

First the connection:

I have an ADSL modem hooked to one of the 3 Ethernet interface on my router. I use rp-pppoe to dial and keep the connection alive.

I have an internal network hooked up to another Ethernet interface on the router. I have about 2-3 PC/Notebook access the internet via the router.

I have a web/SSH server hooked up to another of my Ethernet interface on the router. This is my DMZ. My web and SSH is on standard port because I cant access just any port from work because of the companys firewall. I really wish I could change at least the SSH port.

Access:

On the router, there is basically nothing I can access from the internet directly because all connection requests that comes in is either filtered by the firewall or NAT/DNAT to the DMZ as well as Bittorrent NAT to a specific desktop in the internal network.

In order for me to access the router for emergency situation, I will have to SSH to the server on the DMZ and then SSH back to the router. The router SSH access is password based authentication while the SSH to the DMZ server is key based authentication.

More on what happened:

During the black out Ive mention in the previous post (which happened about 1 day after the missing log entries timestamp). The DMZ server got powered down because that UPS was not able to hold up for the extended period without power but my router UPS survived the black out. That was the reason I was checking the system that lead me to find the gap in the logs.

I know based on the above, I have yet to conclude that my boxes have been hacked and I have yet to restart the DMZ server because I have been busy trying to determine if there is a break in. The next thing I will be doing is to look at the DMZ server to see if there is more evidence that Ive been hacked.

Now, Im building up my tools to go in for forensics. The download is taking some time

Also, I apologize if I seem misleading in the beginning because honestly, I was not thinking what path did the hacker (if there is really one) use to break in to my router. I keep thinking for some reason that the attack was direct to the router. Now that Ive thought about it, if the intrusion was direct to the router, then there must be a security loop hole in the firewall. Chances of that is slim right?

----------

## cynric

I'd imagine what NeddySeagoon meant by rehearsal is disconnecting power to make sure everything appears to work properly. I don't know if the UPS itself has any diagnostic software, but if it does, run that just to make sure everything is tip-top. NeddySeagoon seems to know quite a bit about power (he's helped me once before) and can answer better.

As for the more detailed story, I don't see anything out of place in terms of the physical layout -- it's pretty standard. It seems like you are careful to use both source and destination addresses in your iptable rules, key based ssh authentication, etc. Correlating data from both the firewall and dmz server may help significantly. If nothing else, it should help in creating a timeline or confirm the time of the potential attack. You may want to go ahead and look at earlier logs on one or both systems. This may yield some interesting traffic that was generated while probing your system. Especially if the web server was used since there would be additional logging.

Chances of one machine being hit over another is hard to decide. I'd imagine that the DMZ would be easier to target due to its openness, use of more servers (web and ssh server), etc. Any package that hasn't been updated in a year is a real security risk most likely. If the firewall only filters and routes traffic, it seems like your DMZ would be the best pick. Especially if access to it could lead to "easier" access to the firewall (ssh connection only requires a password). Really it's all speculation until you get enough evidence to piece something together. That'll happen with continued analysis and looking at your network as a whole and not just the firewall itself.

As far as a forensic toolkit, the Helix livecd that didl mentioned is quite nice. It's something that will probably change a lot until you get comfortable with it and learn what tools you need for different situations. One tool that may be useful in pinpointing potential security flaws in a web server is Nikto. Other useful stuff for network auditing are the typical packages like nessus and nmap. SecTools.org has a really good page on auditing tools. Note that these tools don't appear to be on the Helix livecd as it deals with forensic analysis and not network auditing. Although they go hand in hand many times, specialized discs leave out one set of tools in favor of others.

Good luck and keep us posted on your process. I know many people find the process the most difficult part in learning forensics. It's easy to get a list of tools and data. It's hard to learn the best way to use the tools and how to interpret the data you get -- something I'm still working on.

----------

## nosatalian

Occam's Razor: The simplest explanation is probably the most likely.

I'm willing to bet that your power went out for 6 hours.  And the machine booted when it came back.  Were you present when this occured, or out of town?

----------

## weyhan

cynric, NeddySeagoon,

It has been so dense lately that I fail to recognize the simple English word rehearsal It just seems so different seeing the word type out. Haha

Anyway, as Ive mention in my above post that I have had a black out the day after the missing logs and that was a good test on the UPS so Im quite confident with the UPS attached to the router box.

cynric,

Trying to even download all the tools I need to start the investigation is hard enough especially when my internet connection have not been great this few days. I will sure report back here when I have done more investigation. Thanks for the encouragement.

I would also like to thank everyone who has given me encouragement! I really need it.

nosatalian,

In here look at my reply to Akkara. And one more thing, I have a desktop hooked up to the same power source (EDIT: minus the UPS) and was downloading podcast the whole time during the time when the log entries have gone missing. So no power problem.

I'm willing to take you up on your bet, though.   :Very Happy: 

----------

## meyerm

Hi,

perhaps completely unrelated. But could it be a hardware failure (I/O-bus, harddrive, ... )? Having to look after a few DELL servers I already encountered the strangest things  :Wink: . So f.ex. the PERC3 RAID-controller liked to simply hang up itself and no more I/O was possible until the next reboot. The kernel and the loaded services of course continued working until they needed sth. from the harddrive. So perhaps your router - not needing anything from the harddrive - continued to work after some problem which could of course not be logged, after 6 hours wanted to write/read sth. and then got some timeout where the kernel decided to reboot by itself. I would suggest that you start sending out your syslog over the network to another computer so you can always have a look at an uncompromised copy of the log which will also be up to date when the harddrive of the router fails.

M

----------

## NeddySeagoon

weyhan,

Rehearse means to practice. UPS batteries decay with age and need to be replaced from time to time regardless of the level of use.

The only way to determine if your UPS batteries are good enough is to fake a power cut and see what happens.

You may get one of three things:-

1. Everything does an orderly shutdown as you expect - good, it worked.

2. Power goes off when you pull the plug out - bad, your batteries are stone dead.

3. Your UPS batteries fail before the shutdown is complete. This is the worst situation.

Your system goes through a 'brown-out' where the supply voltage is low. What happens now is indeterminate.

It may well include resets as the system Power Good signal cycles between good and bad.

I can add more details if you want but in that case, you still need new UPS batteries,

----------

