# In search of the perfect linux traffic monitor

## njcwotx

I have a couple of projects where I need to monitor web activity and log it and other trafic.

I have used ntop for a number of projects, ntop is great, but it is really just a grand total of all traffic seen since it was last cleared but I need a better history.  Im cool with that in short term situations, but I need something with a little more detail and history.

I need to be able to log traffic and do things like answer questions by managment/hr and my network has several thousand users but almost all of it travels though our DIA.  I can easily setup a mirror port and have done so in the past to troubleshoot.  Now I need a more permanent option, but I cant completely replace the hardware firewalls entirely

Here are some sample questions of the kinds of things we get asked and what we are trying to accomplish.  What percentage of traffic is non-work related, such as how much facebook, twitter, myspace is out there?  Who is going there? and when?  at least by IP/Hostname what was the log from monday 3 weeks ago.

I know there are a lot of tools out there to use, so I wanted to get a discussion goin on what tools people like to use.  Consider the requirements wide-open, it doesnt just have to be a monitor, really I want to try to brainstorm what the tools are to choose from.

----------

## whig

Do you proxy/cache everything with squid? Look at one of its log file analyzers

http://www.squid-cache.org/Scripts/

----------

## idella4

I thought nagios is a tool for network traffic.  It takes alot of setting up.

----------

## njcwotx

Squid would be good, if I didnt have to setup as the proxy but as a passive logger.  Is that possible?

We use nagios here as well, but its more of a polling engine to watch if the network is up and display events.  It doesnt sniff the network and display info on what users are spending 3 hours a day surfing facebook.  Perhaps there is a plugin I have not seen yet.

I have used Ntop as a good sniffer/logger, but I cannot go back 2 weeks and get granular data.  Ntop is great if you want to take an immediate snapshot of a period of time.  I might be able to see some bandwidth usage during that time frame, but I have not seen a way to ask it to report on what user/host X did on Monday of last week.  

An actual example of how I used ntop once was when I was running it I had several users come and complain the internet was 'slow'.  I was able to pull it up and show how in the last few hours their cube neighbors were streaming video, music and dowloading files and eating up the bandwidth rather than working.  After a few weeks of showing my users that any performance issues were related to their nefarious network activities they stopped bugging me.  :Smile:   I just cant dial in a time range and pull u those numbers, its pretty much a 'whats going on now' type of thing.  I would love it if I could dial in a time range and pull up user data.

Browsing Gentoo-portage, there are a ton of packages in net-analyzer but I am not familiar with most of them, and of the ones I do know, those are not really loggers.

++++++++++++++

I am working on a home router though, its currently in building.  Im using a fanless micro pc with an atom processor.  Its pretty cool.  I think Im going to give squid a try there as well as other things.

----------

## njcwotx

http://www.cyberciti.biz/tips/linux-setup-transparent-proxy-squid-howto.html

just found this doc, maybe I could setup the proxy as transparent, however, I would have to make sure it passes muster performance wise.

----------

## whig

squid shouldn't slow things down, most probably the caching will speed things up. There are many workplaces which block all ports except web. Doing that could get a hostile reaction. At the school I worked for the users accepted it. Install a dns proxy/cache too for better effect.

----------

## xibo

I'm using squid for a class c network used by 160-200 clients, it saves nearly 40% of the traffic and replies approximately 60% of the http requests within 10ms ( instead  of 100+ms when requesting them from the original servers ), though it was only like 55% requests and 10% traffic before i exemplary denied access to some people running downloaders the whole day and waited for them to spread the word that it's not welcome.

I leave it to your imagination to decide by how much 40% less traffic makes things faster...

Nevertheless tcpdump and its friends are still needed, because there's way too many broken webservers and clients that require firewall entries for you net to operate properly.

I.e. the day before yesterday my squid-3.1.5.1 crashed after getting requested http://imgcdn.pandora.tv/pan_img/KMP/Download/kmpver.txt multiple times per second for 7 hours, thanks to pandora's faulty http headers.

Not to mention that there are still people alive that consider the internet more then just the web...

----------

