# kvm + br0 + tap0 + eth0 = virtual hub? [yep!]

## vaxbrat

I've been banging my head against this trying to do something that I thought would be fairly common.  I'm trying to get a kvm virtual machine (windows-xp) to use its host as a virtual hub to the outside world.  I want this vm to be in the same subnet as the host as if it had it's own nic and connection to the outside world.  It needs to get a dhcp address from an external server as if it were just another drop on the network switch.  Meanwhile the host needs to maintain its network identity.  I don't care about filtering with either iptables or ebtables (yet).  

br0 consists of eth0 and tap0 (a tuntap tap device) in promiscuous mode with no ip.  br0 is set to 192.168.11.12 while the windows vm thinks it's supposed to be 192.168.11.200

I'm thinking arp replies back to the vm are getting thrown on the floor because I see its broadcasts coming out of tap0 in wireshark but nothing going back.  ping of br0 itself from the vm yields an arp request from tap0 asking for who owns 192.168.11.12 but nothing else.  Ping from br0 to 192.168.11.200 works.  ping to an outside ip yields a similar arp request on tap0 but with no reply

It looks like some mac shenanigans with ebtables is needed, but all of the docs out there are ugly.  Everybody assumes your setup in bridging docs consists of three ethernet devices where eth0 is outside and an eth1 and eth2 are bridging a lan segment.  Not what I need.

Current software on amd64 stable.

kernel 2.6.27-gentoo-r8

kvm-84 (unmasked from ~amd64)

ebtables-2.0.8.2-r2

bridge-utils-1.4

iptables-1.4.2-r2

shorewall-common-4.0.15

shorewall-perl-4.0.15

/proc/sys/net/ipv4/ip_forward is set to 1

/proc/sys/net/bridge/* entries are all set to 1

Even though I have the filtering stuff installed, I'm totally passthru at the moment:

```
# iptables -L

Chain INPUT (policy ACCEPT)

target     prot opt source               destination

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination

# ebtables -L; ebtables -t nat -L; ebtables -t broute -L

Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

Bridge table: nat

Bridge chain: PREROUTING, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT

Bridge table: broute

Bridge chain: BROUTING, entries: 0, policy: ACCEPT

```

I've played with vmware server and the bridge vmnet setup before so I know this is doable.  I would use their vmnet stuff for this if I knew how to snake my way into their taps.Last edited by vaxbrat on Tue Mar 24, 2009 4:22 am; edited 1 time in total

----------

## vaxbrat

Adding a host route on the host system allows the outside world to ping in.  The outside world's arp request gets over to tap0 fine and the vm responds back so that the pings can then happen:

```
route add -host 192.168.11.200 dev tap0

```

----------

## vaxbrat

The previous rock didn't quite get things done.  This route statement  on the host will finally allow transparent two way traffic in and out of the guest.  I still need to try this in a dhcp setup where the guest gets its address from another server.

```
route add -host 192.168.11.200 gw 192.168.11.12 dev br0
```

This whole process has caused much more hair pulling than I had originally bargained for   :Confused: 

----------

## vaxbrat

It always helps to start from a fresh boot after you have been chasing your tail around and around.

----------

## vaxbrat

I have a HP desktop at work with the VPro chipset.  I have yet to duplicate at work what I had working at home with the Amd Phenom and host route.  Not sure if there is something screwed up with the bridge or because of the fact that I'm playing with dhcp from an outside server, but I have yet to successfully ping inside and out of the guest.

----------

## vaxbrat

I put a Fedora 10 install on the work box in a double boot and then will try a vmware server 2 with its bridge scheme to see what happens.  It was interesting to see that adding a vmware server 2 setup to the gentoo side still didn't get the networking to work.  There must be something funny in the kernel with iptables/ebtables and whatever vmware wants to do.

----------

## vaxbrat

One thing that I just noticed that was screwing me up is the conception that I would have the same mac address on the hypervisor's tap interface as what the guest sees on its virtual nic.  I had one system where both were the same and was unable to ping back into the guest from an external host or to ping anything but the hypervisor from the guest.  Another system where the two were different allowed me all the way in and out.

Thus, the /etc/conf.d/net has 

```
mac_tap0="52:54:00:12:34:56"
```

while the kvm command line has the snippet

```
kvm -net nic,macaddr=00:1d:92:ab:3f:79
```

One of those is the mac range assigned to the realtek nic that the guest thinks it sees while the other is a range I had seen from kvm examples out there

I'm putting both in the /etc/ethers but wonder if I really need this file now

```
00:1d:92:ab:3f:79       192.168.11.202

52:54:00:12:34:58       192.168.11.202
```

----------

## vaxbrat

I fixed a work box today to go back to using kvm after working with a vmware server 2 set up.  The inside/outside mac address scheme for the tap is the key.  /etc/ethers is not needed.  I don't think /etc/sysctl.conf is crucial either but making sure that ip_forward is set with that is probably a good thing anyway:

```
# /etc/sysctl.conf

#

# For more information on how this file works, please see

# the manpages sysctl(8) and sysctl.conf(5).

#

# In order for this file to work properly, you must first

# enable 'Sysctl support' in the kernel.

#

# Look in /proc/sys/ for all the things you can setup.

#

# enables packet forwarding

net.ipv4.ip_forward = 1

# enables IP dynaddr

net.ipv4.ip_dynaddr = 1

# Disable ECN

#net.ipv4.tcp_ecn = 0

# Enables source route verification

net.ipv4.conf.default.rp_filter = 1

# Enable reverse path

net.ipv4.conf.all.rp_filter = 1

# Enable SYN cookies (yum!)

# http://cr.yp.to/syncookies.html

#net.ipv4.tcp_syncookies = 1

# Disable source route

#net.ipv4.conf.all.accept_source_route = 0

#net.ipv4.conf.default.accept_source_route = 0

# Disable redirects

#net.ipv4.conf.all.accept_redirects = 0

#net.ipv4.conf.default.accept_redirects = 0

# Disable secure redirects

#net.ipv4.conf.all.secure_redirects = 0

#net.ipv4.conf.default.secure_redirects = 0

# Ignore ICMP broadcasts

#net.ipv4.icmp_echo_ignore_broadcasts = 1

# Disables the magic-sysrq key

#kernel.sysrq = 0

# When the kernel panics, automatically reboot in 3 seconds

#kernel.panic = 3

# Allow for more PIDs (cool factor!); may break some programs

#kernel.pid_max = 999999

# You should compile nfsd into the kernel or add it

# to modules.autoload for this to work properly

# TCP Port for lock manager

#fs.nfs.nlm_tcpport = 0

# UDP Port for lock manager

#fs.nfs.nlm_udpport = 0

```

If you make edits and want the running system to change do this afterwards:

```
sysctl -p
```

It would be interesting to see whether the vmware "vmnet" network devices as well as its hidden virtual bridge device to your nic (ie eth0) could co-exist with a br0-eth0-tap0 style bridge on the same nic.  While doing the vmware testing, I had the /etc/conf.d/net files swapped so that only eth0 was up and running.

----------

## vaxbrat

I should note that the host route doesn't need to be added to the scheme either.  That windows guest at work uses dhcp from an external host server and works fine without a host route.  In fact, the corporate lan is based on a class B block which is subnetted into class C pools.  Random assignment gave my hypervisor an address from one class C block while the guest got one from another.  That didn't impact the bridge pasthru.

----------

## vaxbrat

I updated the kvm entry on the gentoo wiki with this approach:

http://en.gentoo-wiki.com/wiki/KVM#Enabling_the_access_to_Internet

----------

## jago25_98

I was kind of hoping it would be possible to just link all interfaces into a bridge and then just firewall the bridge - much simpler! 

Personally all I need to do is join Tun/Tap to eth0(internet side).

My config file worked initially but then it stopped after about 5 hours and I've no idea why:

```
 # Used by ifup(8) and ifdown(8). See the interfaces(5) manpage or

# /usr/share/doc/ifupdown/examples for more information.

# The loopback network interface

auto lo

iface lo inet loopback

# The primary network interface

# Uncomment this and configure after the system has booted for the first time

auto br0

iface br0 inet static

pre-up ifconfig eth0 up

address myIP.to.internet.side

netmask 255.255.255.0

gateway 67.23.25.1

dns-nameservers 72.3.128.240 72.3.128.241

bridge_ports eth0

bridge_fd 9

bridge_hello 2

bridge_maxage 12

bridge_stp off

auto eth1

iface eth1 inet static

address myIP.to.slicehost.side.i.think

netmask 255.255.224.0

auto tap0

iface tap0 inet static

pre-up tunctl -u j -t tap0

address 10.0.0.1

netmask 255.0.0.0

#auto br0

# iface br0 inet dhcp

# bridge_ports eth0 eth1 tun0 tun1

up route add -net 10.176.0.0 netmask 255.248.0.0 gw 10.176.96.1

down route add -net 10.176.0.0 netmask 255.248.0.0 gw 10.176.96.1

up route add -net 10.191.192.0 netmask 255.255.192.0 gw 10.176.96.1

down route add -net 10.191.192.0 netmask 255.255.192.0 gw 10.176.96.1 
```

edit: looks like gentoo doesn't use the same network config file as me anymore, it's been a while

----------

## vaxbrat

I've had some of these bridges up for days at a time so not sure why you are losing it after 5 hours.

Also if you bridge everything to eth0, you should be able to put firewall rules on eth0 to have global effect.  Likewise, you should be able to put individual rules at the taps to customize for each guest.

----------

