# Incoming net traffic permanent cache

## wmigda

Hi,

maybe the question I'd like to ask is rather gentoo-unrelated but being a networking noob I don't seem to find a solution myself (even by thinking of names of boxes to google for), and I know this forum has offered valuable responses many times, so I'll give it a try.

Anyway, I'd like to find a way to be able to have a permanent cache storage on local desktop computer for data received from selected ip adresses (be it a range or a mask). The ideal solution would be to have the entire incoming traffic being passed through some kind of a sw box, which will filter files received from sources of interest, and store them for future reuse. Then the original application (which unfortunately hasn't got a permanent cache) would be able to use cached resourced instead of downloading them over the wire. I thought about having a local proxy (I don't even know how it would have to be done  :Sad:  ), but I guess it might be equivalent to using a gun to kill a fly.

TIA,

Wojtek

----------

## nixnut

You may want to take a look at net-analyzer/tcpdump

----------

## wmigda

Thank you, but looking at its manpage it says that it operates on a packet level. Is it possible to use tcpdump on a url level, or a different tools should be used ?

I'm looking for sth, that being setup to cache data from foo.com, then when a given application downloads http://foo.com/asset/zoo.bar it will be cached. The next time the app is started and attempts to download the same asset, then first it will be checked for in the cache before attempt to go beyond the local machine.

-Wojtek

*** edit ***

How about squid ? Will it do ?

----------

## boerKrelis

I do not see how this could work at the TCP/IP level.

However, there are various protocol-specific solutions available. For HTTP, use an HTTP proxy. For NFS, have a look at FS-Cache. For email, you could run net-mail/offlineimap .

If you simply to cache asset http://foo.com/asset/zoo.bar, working with Squid may or may not work. It depends on how foo.com is configured. To see if zoo.bar is still fresh in its cache, Squid will ask foo.com some questions and if foo.com answers with a NOT MODIFIED header, then Squid will consider zoo.bar fresh in its cache. So what will happen if foo.com does not support answering of squid's questions? Zoo.bar will be redownloaded, even if it hasn't changed!

So what you might want to do is to manually cache zoo.bar. If all you ever need from foo.com is zoo.bar, then you could add

```
127.0.0.1 foo.com
```

 to your /etc/hosts, run something like www-servers/webfs to serve '/asset/zoo.bar', and do a manual fetch every now and then.

If you'd like to access other stuff at foo.com you'll need something more flexible than www-servers/webfs, linke www-servers/nginx. Nginx can be configured to proxy foo.com except for /asset/zoo.bar.Last edited by boerKrelis on Sat Jan 09, 2010 1:04 pm; edited 1 time in total

----------

## wmigda

 *boerKrelis wrote:*   

> For HTTP, use an HTTP proxy.

 

Thanks boerKrelis.

Yes, I'm interested in http, however, the application is flash-based, and I don't know: 1. whether it is possible to tell flash player to connect through a proxy, 2. if I can use http proxy (such as squid) on the same computer I wher I use this flash application ?

----------

## boerKrelis

I think that webpage-embedded flash apps do not implement HTTP, they just ask the host browser to fetch resource $URL. So they honor the webbrowser's settings.

You can certainly use a http proxy on the same machine as you use to browse the net with. I do it all the time (I use net-proxy/polipo + net-proxy/adzapper to filter ads). For simple usage scenarios, I'd recommend polipo over squid as running it can be as simple as opening up a terminal and issuing 'polipo'.

Did you read the bit in my previous reply about not running a caching proxy, but redirecting requests to foo.com to some internal webserver? I was editing my post while you were posting your reply.

----------

## pigeon768

What is the application? squid can be configured to cache, for instance, youtube videos in addition to standard html pages.

----------

## Hu

Even if the program does not honor proxy settings, the right iptables invocation can be used to force the connection to go through a transparent local proxy.  This is more complicated, and not all HTTP proxies are capable of serving as transparent proxies.  I think Squid can do it, but I have never done so.

----------

## boerKrelis

Squid can do it. In two ways, in fact. With Squid 3.1+ and netfilter's TPROXY you can even spoof the IP of the website the browser is connecting to. I've only done the non-spoofing type of interception, though.

----------

