# find source of attempted outgoing blocked byfirewall[solved]

## huuan

Our server has a number oif virtual hosts and has the firewall blocking outgoing packets on everything except to specific hosts, e.g., rsync mirrors and the like.

Recently I noticed in the firewall logs that there are repeated attempts, every few hours, to communicate with another host outgoing  via port 3306, presumably the other host is a mysql server of some knid. The other host has all ports filtered and it's ip does not resolve by DNS lookup so there's no clue there.

Trying to find the source of these outgoing packets I enabled uid logging and saw that apache was the uid.

My own tests indicate that such outgoing can be produced by scripting such as php, of course phpmyadmin immediately springs to mind as a source but without logs that track such things I can't really tell.

I can't see a way to use apache logs to track outgoing requests, only incoming, and there's no clue in the error logs even though the output is blocked by the firewall.  The frequency of the attempts leads me to think it's a script as they occur even in the wee hours.

Any ideas as to how to track down the source of these packets?

Thanks.Last edited by huuan on Sat May 03, 2008 1:43 am; edited 1 time in total

----------

## Hu

If you are willing to risk a data leak, you could allow it to connect out once and run tcpdump to capture what is sent and received.  That may give you some hints as to what is initiating it.  Caution: depending on your authority relative to other users on the box, you may run into legal problems if you sniff their traffic without consent.  If you are not sure whether such sniffing is legal in your jurisdiction, consult your legal department for advice.

Another option would be to attach strace to all the Apache processes and log calls to fork/exec.  Depending on your site and load, this may not be feasible due to the volume of output produced.  If it is feasible, you could use the strace logs to identify what processes were running at the time of the connection attempt.

----------

## huuan

Hello Hu,

Thanks for your help.

I was hoping for something slightly simpler..oh well.

I'm unable to leak the data but the server is low traffic so I may be able to do as you suggest attach strace to all the apache processes and log calls to fork/exec. 

Looks like i need to do 

# strace -t -e trace=process -o stracelog -p pid -p pid2 -p pid3 etc etc

OK I tried execve , network, process, fork but they were either too quiet or too noisy to be of use

trace=signal seems to give pertinent data that relates well to my own attempts to pass the firewall, here's an example output as I use a script to pass the firewall:

290  00:19:01 rt_sigaction(SIGPROF, {0x5101d75f, [PROF], SA_RESTART}, {0x5101d75f, [PROF], SA_RESTART}, 8) = 0

2290  00:19:01 rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0

2290  00:19:01 rt_sigaction(SIGPROF, {0x5101d75f, [PROF], SA_RESTART}, {0x5101d75f, [PROF], SA_RESTART}, 8) = 0

2290  00:19:01 rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0

2860  00:20:13 rt_sigaction(SIGPROF, {0x5101d75f, [PROF], SA_RESTART}, {0x5101d75f, [PROF], SA_RESTART}, 8) = 0

sadly I don't have much clue about the data it is putting out so without lots more research I'm stalled.

----------

## huuan

Another approach occurred to me but I'm not that knowledgeable in the area:

grsec is enabled but I have a bunch of the logging disabled as it was producing HUGE logs in a very short time.

OK I've re-enabled these two and will see what the logs show around when the next dropped output is:

audit_chdir      exec_logging

----------

## timeBandit

 *huuan wrote:*   

> The other host has all ports filtered and it's ip does not resolve by DNS lookup so there's no clue there.

 

Instead of nslookup or host, try whois ip-address to find the ISP that owns the address. By itself it tells you nothing about the source process, but the name of the address range owner might suggest additional lines of investigation.

----------

## Hu

The SIGPROF activity looks like background noise to me.  I notice you did not include -f in your strace command, so you did not trace into the offending process at all.  A quick test shows that -e trace=process will log the fork call in the parent, but will not log the exec call in the child.  If the offending code is not run inside the main Apache process through something like mod_php or mod_perl, then it will be in a separate process and you will need to have strace follow forks to catch it.  I suggest using strace -f -tt -e trace=process,network -o log.strace.  Also, try using strace to monitor a netcat which attempts a prohibited connection.  That will give you a baseline for how the offending process will look when you find it.  Once you have a baseline, you can grep the Apache strace instead of searching it by hand.  That should largely mitigate the difficulties associated with having a file that is "too noisy."

----------

## huuan

Thanks Hu,

those are all great tips which I will add to my arsenal.

I've now found the source of the problem. One of the virtual hosts is running Coppermine photo album and although a fairly recent version, about 2 months old, it had an sql injection vulnerability.  

Re-enabling audit_chdir in grsec helped me track the output. I could see which virtual host was doing stuff right before the attempted output and that led me to see that there was an ip (also that did not resolve via hostx) that was accessing the coppermine album immediately before the attempts to talk to the remote mysql server.

 I then googled "coppermine security" and found the reports of the security flaw. 

Then I looked in the coppermine directory using

ls -rlt

to see which were the most recent files updated and there it was: the coppermine config file had been completely highjacked to the ip I had been searching for but interestingly via a hostname which when I hostx'd  resolved to the ip I had been searching for. I should also mention for timeBandits sake that I had already previously used tools such as samspade  and various other whois services to check out adjacent hosts without getting too much clue. Once I have cleaned everything up I may be more forthright about that.

So now as well as fixing that I need to pay more attention and put up a security  app that watches files for changes.

Thanks so much for your help, especially Hu but all others as well.

I will be back to give a full report later when I have patched things up.

----------

## huuan

OK it was a bad install of coppermine on a subdomain. The install.php file was still there and usable after the install so someone just used it over and redirected the album to their site. The site had a very old copy of coppermine that had been compromised.

----------

