# [SOLVED] KVM and network bridge

## AgBr

I need help with the setup of network configuration of my Host system.

Goal is to set up a virtual hub so that the guests are on the same subnet as the host and are visible to other hosts on the net as if they were physical machines.

I tried to follow this instruction. To my opinion the instruction is somewhat ambiguous as there are two concepts (virtual hub and NAT/masquerading) mixed up. May be I didn't understand it all. So I need a little help here.

Up till now I have one FreeBSD guest (elrond) running, to which I have access from the host (dragon) via ssh. From the guest I can see the host but can not see any other machine on the net (featuring celeborn as an example below) as well as I can not get into the guest from any other machine except from the virtualization host.

My configuration so far looks like this:

```

dragon ~ # ifconfig eth0

eth0      Protokoll:Ethernet  Hardware Adresse 00:19:99:b9:8d:2d  

          inet6 Adresse: fe80::219:99ff:feb9:8d2d/64 Gültigkeitsbereich:Verbindung

          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:45111 errors:0 dropped:0 overruns:0 frame:0

          TX packets:4828 errors:0 dropped:0 overruns:0 carrier:0

          Kollisionen:0 Sendewarteschlangenlänge:1000 

          RX bytes:6556353 (6.2 MiB)  TX bytes:472331 (461.2 KiB)

          Interrupt:18 Speicher:cd420000-cd440000 

dragon ~ # ifconfig br0

br0       Protokoll:Ethernet  Hardware Adresse 00:00:00:00:00:02  

          inet Adresse:172.16.1.4  Bcast:172.16.31.255  Maske:255.255.224.0

          inet6 Adresse: fe80::200:ff:fe00:2/64 Gültigkeitsbereich:Verbindung

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:23760 errors:0 dropped:0 overruns:0 frame:0

          TX packets:3716 errors:0 dropped:0 overruns:0 carrier:0

          Kollisionen:0 Sendewarteschlangenlänge:0 

          RX bytes:3601507 (3.4 MiB)  TX bytes:368598 (359.9 KiB)

dragon ~ # ifconfig tap0

tap0      Protokoll:Ethernet  Hardware Adresse 00:00:00:00:00:02  

          UP BROADCAST PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:1 overruns:0 carrier:0

          Kollisionen:0 Sendewarteschlangenlänge:500 

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

dragon ~ # ifconfig tap1

tap1      Protokoll:Ethernet  Hardware Adresse 00:00:00:00:00:03  

          inet6 Adresse: fe80::200:ff:fe00:3/64 Gültigkeitsbereich:Verbindung

          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:444 errors:0 dropped:0 overruns:0 frame:0

          TX packets:18557 errors:0 dropped:1 overruns:0 carrier:0

          Kollisionen:0 Sendewarteschlangenlänge:500 

          RX bytes:38115 (37.2 KiB)  TX bytes:2090768 (1.9 MiB)

dragon ~ # brctl show br0

bridge name     bridge id               STP enabled     interfaces

br0             8000.000000000002       no              eth0

                                                        tap0

                                                        tap1

        

dragon ~ # sysctl net.ipv4.conf.eth0

net.ipv4.conf.eth0.forwarding = 0

net.ipv4.conf.eth0.mc_forwarding = 0

net.ipv4.conf.eth0.accept_redirects = 1

net.ipv4.conf.eth0.secure_redirects = 1

net.ipv4.conf.eth0.shared_media = 1

net.ipv4.conf.eth0.rp_filter = 1

net.ipv4.conf.eth0.send_redirects = 1

net.ipv4.conf.eth0.accept_source_route = 1

net.ipv4.conf.eth0.accept_local = 0

net.ipv4.conf.eth0.src_valid_mark = 0

net.ipv4.conf.eth0.proxy_arp = 0

net.ipv4.conf.eth0.medium_id = 0

net.ipv4.conf.eth0.bootp_relay = 0

net.ipv4.conf.eth0.log_martians = 0

net.ipv4.conf.eth0.tag = 0

net.ipv4.conf.eth0.arp_filter = 0

net.ipv4.conf.eth0.arp_announce = 0

net.ipv4.conf.eth0.arp_ignore = 0

net.ipv4.conf.eth0.arp_accept = 0

net.ipv4.conf.eth0.arp_notify = 0

net.ipv4.conf.eth0.proxy_arp_pvlan = 0

net.ipv4.conf.eth0.disable_xfrm = 0

net.ipv4.conf.eth0.disable_policy = 0

net.ipv4.conf.eth0.force_igmp_version = 0

net.ipv4.conf.eth0.promote_secondaries = 0

dragon ~ # sysctl net.ipv4.conf.br0

net.ipv4.conf.br0.forwarding = 0

net.ipv4.conf.br0.mc_forwarding = 0

net.ipv4.conf.br0.accept_redirects = 1

net.ipv4.conf.br0.secure_redirects = 1

net.ipv4.conf.br0.shared_media = 1

net.ipv4.conf.br0.rp_filter = 1

net.ipv4.conf.br0.send_redirects = 1

net.ipv4.conf.br0.accept_source_route = 1

net.ipv4.conf.br0.accept_local = 0

net.ipv4.conf.br0.src_valid_mark = 0

net.ipv4.conf.br0.proxy_arp = 1

net.ipv4.conf.br0.medium_id = 0

net.ipv4.conf.br0.bootp_relay = 0

net.ipv4.conf.br0.log_martians = 0

net.ipv4.conf.br0.tag = 0

net.ipv4.conf.br0.arp_filter = 0

net.ipv4.conf.br0.arp_announce = 1

net.ipv4.conf.br0.arp_ignore = 0

net.ipv4.conf.br0.arp_accept = 0

net.ipv4.conf.br0.arp_notify = 0

net.ipv4.conf.br0.proxy_arp_pvlan = 0

net.ipv4.conf.br0.disable_xfrm = 0

net.ipv4.conf.br0.disable_policy = 0

net.ipv4.conf.br0.force_igmp_version = 0

net.ipv4.conf.br0.promote_secondaries = 0

dragon ~ # sysctl net.ipv4.conf.tap0

net.ipv4.conf.tap0.forwarding = 0

net.ipv4.conf.tap0.mc_forwarding = 0

net.ipv4.conf.tap0.accept_redirects = 1

net.ipv4.conf.tap0.secure_redirects = 1

net.ipv4.conf.tap0.shared_media = 1

net.ipv4.conf.tap0.rp_filter = 1

net.ipv4.conf.tap0.send_redirects = 1

net.ipv4.conf.tap0.accept_source_route = 1

net.ipv4.conf.tap0.accept_local = 0

net.ipv4.conf.tap0.src_valid_mark = 0

net.ipv4.conf.tap0.proxy_arp = 0

net.ipv4.conf.tap0.medium_id = 0

net.ipv4.conf.tap0.bootp_relay = 0

net.ipv4.conf.tap0.log_martians = 0

net.ipv4.conf.tap0.tag = 0

net.ipv4.conf.tap0.arp_filter = 0

net.ipv4.conf.tap0.arp_announce = 0

net.ipv4.conf.tap0.arp_ignore = 0

net.ipv4.conf.tap0.arp_accept = 0

net.ipv4.conf.tap0.arp_notify = 0

net.ipv4.conf.tap0.proxy_arp_pvlan = 0

net.ipv4.conf.tap0.disable_xfrm = 0

net.ipv4.conf.tap0.disable_policy = 0

net.ipv4.conf.tap0.force_igmp_version = 0

net.ipv4.conf.tap0.promote_secondaries = 0

dragon ~ # sysctl net.ipv4.conf.tap1

net.ipv4.conf.tap1.forwarding = 0

net.ipv4.conf.tap1.mc_forwarding = 0

net.ipv4.conf.tap1.accept_redirects = 1

net.ipv4.conf.tap1.secure_redirects = 1

net.ipv4.conf.tap1.shared_media = 1

net.ipv4.conf.tap1.rp_filter = 1

net.ipv4.conf.tap1.send_redirects = 1

net.ipv4.conf.tap1.accept_source_route = 1

net.ipv4.conf.tap1.accept_local = 0

net.ipv4.conf.tap1.src_valid_mark = 0

net.ipv4.conf.tap1.proxy_arp = 0

net.ipv4.conf.tap1.medium_id = 0

net.ipv4.conf.tap1.bootp_relay = 0

net.ipv4.conf.tap1.log_martians = 0

net.ipv4.conf.tap1.tag = 0

net.ipv4.conf.tap1.arp_filter = 0

net.ipv4.conf.tap1.arp_announce = 0

net.ipv4.conf.tap1.arp_ignore = 0

net.ipv4.conf.tap1.arp_accept = 0

net.ipv4.conf.tap1.arp_notify = 0

net.ipv4.conf.tap1.proxy_arp_pvlan = 0

net.ipv4.conf.tap1.disable_xfrm = 0

net.ipv4.conf.tap1.disable_policy = 0

net.ipv4.conf.tap1.force_igmp_version = 0

net.ipv4.conf.tap1.promote_secondaries = 0

dragon ~ # sysctl net.bridge

net.bridge.bridge-nf-call-arptables = 0

net.bridge.bridge-nf-call-iptables = 0

net.bridge.bridge-nf-call-ip6tables = 0

net.bridge.bridge-nf-filter-vlan-tagged = 0

net.bridge.bridge-nf-filter-pppoe-tagged = 0

dragon ~ # lsmod

Module                  Size  Used by

iptable_filter           984  0 

ip_tables               7681  1 iptable_filter

x_tables                9464  2 iptable_filter,ip_tables

kvm_intel             107912  3 

kvm                   243019  1 kvm_intel

tun                     9802  5 

bridge                 52516  0 

ipv6                  205677  17 bridge

stp                     1051  1 bridge

llc                     2538  2 bridge,stp

r8169                  34883  0 

mptsas                 27327  1 

8250_pnp                3391  0 

mptscsih               14369  1 mptsas

i7core_edac            12552  0 

i2c_i801                6174  0 

edac_core              24556  3 i7core_edac

mptbase                44384  2 mptsas,mptscsih

serio_raw               2913  0 

mii                     2823  1 r8169

ac                      2373  0 

e1000e                107805  0 

8250                   15537  1 8250_pnp

serial_core            12029  1 8250

dragon ~ # netstat -rn

Kernel IP Routentabelle

Ziel            Router          Genmask         Flags   MSS Fenster irtt Iface

0.0.0.0         172.16.20.5     0.0.0.0         UG        0 0          0 br0

127.0.0.0       -               255.0.0.0       !         - -          - -

172.16.0.0      0.0.0.0         255.255.224.0   U         0 0          0 br0

```

As you can see from the arp-Tables, arp seems to work

```

dragon ~ # arp -a

celeborn (172.16.1.2) auf 00:19:99:7d:fa:78 [ether] auf br0

elrond (172.16.1.7) auf 00:00:00:00:00:03 [ether] auf br0

celeborn# arp -a|grep 172.16.1.7

? (172.16.1.7) at 00:00:00:00:00:03 on bge0 expires in 1146 seconds [ethernet]

celeborn# arp -a | grep 172.16.1.4

dragon (172.16.1.4) at 00:00:00:00:00:02 on bge0 expires in 1087 seconds [ethernet]

elrond# arp -a

? (172.16.1.4) at 00:00:00:00:00:02 on re0 expires in 907 seconds [ethernet]

? (172.16.1.7) at 00:00:00:00:00:03 on re0 permanent [ethernet]

? (172.16.1.2) at 00:19:99:7d:fa:78 on re0 expires in 1115 seconds [ethernet]

 

```

The VM is attached to tap1. tap0 isn't in use at this point. Interestingly br0 has the same MAC as tap0. 

The guest is started with

```

qemu-kvm -hda /usr/local/util/freebsd.img -hdb /dev/sdb \

                -net nic,macaddr=00:00:00:00:00:03 -net tap,ifname=tap1,script=no,downscript=no \

                -cdrom /dev/sr0 -curses -k de 2

```

I have tried fiddeling with proxy_arp and forwarding to no avail. As far as I understand at least forwarding should not matter here, as arp shows the tap Interfaces outside the host-Machine (on celeborn). 

Iptables isn't involved either:

```

dragon ~ # iptables -L

Chain INPUT (policy ACCEPT)

target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination 

```

So what am I missing here?Last edited by AgBr on Wed Jun 06, 2012 8:46 am; edited 1 time in total

----------

## Hu

Please post a command that fails to connect in the way you want, and the full output generated by that command.  Also post the output of /sbin/ip route or the equivalent for both endpoints of the attempted connection.

----------

## AgBr

 *Hu wrote:*   

> Please post a command that fails to connect in the way you want, and the full output generated by that command.  Also post the output of /sbin/ip route or the equivalent for both endpoints of the attempted connection.

 

```

user@gandalf ~ $ ssh root@172.16.1.7

ssh: connect to host 172.16.1.7 port 22: Connection timed out

user@gandalf ~ $ ping 172.16.1.7

PING 172.16.1.7 (172.16.1.7) 56(84) bytes of data.

Ping doesn't return anything ^C after quite some while. 

user@gandalf ~ $ netstat -rn

Kernel IP Routentabelle

Ziel            Router          Genmask         Flags   MSS Fenster irtt Iface

172.16.0.0      0.0.0.0         255.255.224.0   U         0 0          0 eth0

127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo

0.0.0.0         172.16.20.5     0.0.0.0         UG        0 0          0 eth0

user@gandalf ~ $ /sbin/arp -a

dragon (172.16.1.4) auf 00:00:00:00:00:02 [ether] auf eth0

celeborn (172.16.1.2) auf 00:19:99:7d:fa:78 [ether] auf eth0

? (172.16.1.7) auf 00:00:00:00:00:03 [ether] auf eth0

ulli@gandalf ~ $ 

```

Logged into the guest (elrond) via the host (dragon) and ping to gandalf (172.16.20.9)

```

elrond# ping 172.16.20.9

PING 172.16.20.9 (172.16.20.9): 56 data bytes

^C

--- 172.16.20.9 ping statistics ---

60 packets transmitted, 0 packets received, 100.0% packet loss

elrond# netstat -rn

Routing tables

Internet:

Destination        Gateway            Flags    Refs      Use  Netif Expire

default            172.16.20.5        UGS         0        0    re0

127.0.0.1          link#3             UH          0       25    lo0

172.16.0.0/19      link#1             U           0     1385    re0

172.16.1.7         link#1             UHS         0        0    lo0

Internet6:

Destination                       Gateway                       Flags      Netif Expire

::/96                             ::1                           UGRS        lo0

::1                               ::1                           UH          lo0

::ffff:0.0.0.0/96                 ::1                           UGRS        lo0

fe80::/10                         ::1                           UGRS        lo0

fe80::%re0/64                     link#1                        U           re0

fe80::200:ff:fe00:3%re0           link#1                        UHS         lo0

fe80::%lo0/64                     link#3                        U           lo0

fe80::1%lo0                       link#3                        UHS         lo0

ff01::%re0/32                     fe80::200:ff:fe00:3%re0       U           re0

ff01::%lo0/32                     ::1                           U           lo0

ff02::/16                         ::1                           UGRS        lo0

ff02::%re0/32                     fe80::200:ff:fe00:3%re0       U           re0

ff02::%lo0/32                     ::1                           U           lo0

elrond# arp -a

? (172.16.20.9) at 00:30:05:40:68:aa on re0 expires in 1042 seconds [ethernet]

? (172.16.1.4) at 00:00:00:00:00:02 on re0 expires in 1061 seconds [ethernet]

? (172.16.1.7) at 00:00:00:00:00:03 on re0 permanent [ethernet]

? (172.16.1.2) at 00:19:99:7d:fa:78 on re0 expires in 1141 seconds [ethernet]

```

Arp-Tables presented again for your convenience. As you can see, arp does work. If arp works, the theory is, that layer 2 must work. Otherwise the machines would not have any means to know the mac-addresses for the target-interfaces of the respective IP-addresses. If layer 2 works, on the local net IP-traffic  should work too as there is no router involved. I can log into the guest (172.16.1.7/19) from the host (172.16.1.4) but not from any other machine on the same subnet (172.16.20.9/19 or 172.16.1.2/19) for instance)

----------

## AngelKnight

You didn't mention whether or not you turned forwarding on on the host, with

```
sysctl net.ipv4.ip_forward=1
```

----------

## AgBr

 *AngelKnight wrote:*   

> You didn't mention whether or not you turned forwarding on on the host, with
> 
> ```
> sysctl net.ipv4.ip_forward=1
> ```
> ...

 

I have tried this too despite that it should not be necessary. It did not make a difference.

----------

## AngelKnight

Argh, yes you're correct: there's no L3 forwarding here.  Sorry, I thought I'd read more carefully.

If I read this correctly, you set the host's tap1 to be a forced MAC (0000.0000.0003) -and- set the inside guest to use the same forced MAC.

If you did, don't: let the host's tap interface float and use qemu-kvm commandline to force the MAC on the adapter inside.  I've already tested this; when the host TAP has the same MAC as the guest's simulated ethernet, the host TAP will tend to eat the frames, not pass them through.  Don't worry about collisions too much: the autoassigned address has most-sig-byte's bit 1 set (marking it local-admin).

I vaguely remember being bit by this but evidently misremembered why earlier, sorry about that.

----------

## AgBr

 *AngelKnight wrote:*   

> Argh, yes you're correct: there's no L3 forwarding here.  Sorry, I thought I'd read more carefully.
> 
> If I read this correctly, you set the host's tap1 to be a forced MAC (0000.0000.0003) -and- set the inside guest to use the same forced MAC.
> 
> If you did, don't: let the host's tap interface float and use qemu-kvm commandline to force the MAC on the adapter inside.  I've already tested this; when the host TAP has the same MAC as the guest's simulated ethernet, the host TAP will tend to eat the frames, not pass them through.  Don't worry about collisions too much: the autoassigned address has most-sig-byte's bit 1 set (marking it local-admin).
> ...

 

I was following the above mentioned instructions here but I will try that. Meanwhile I have found out by contemplating some tcpdump output, that all broadcast frames go in through the tap interface and all direct frames from the guest and all broadcasts go out through the tap but not through the bridge-Interface. So all frames having specific MAC-Addresses coming from the tap are blocked by the bridge interface. 

The box in question is remote. I hope fiddeling with the tap interface will not kill the bridge. Otherwise I will have to wait until tomorrow.

----------

## AgBr

 *AngelKnight wrote:*   

> 
> 
> ... let the host's tap interface float and use qemu-kvm commandline to force the MAC on the adapter inside. 

 

The problem is solved this way. Thank you. The wiki seems to be wrong in this respect.

----------

## AngelKnight

This wiki specifically says to make the host machine the defaultrouter in the guest, not the actual network's bridge.  This is probably why.

Trying to use the real network segment's defaultrouter will not work for this configuration as you have discovered.

This wiki needs quite a bit of fixing around networking actually.  Ugh.  I'm too drunk to fix this now.  Anyone?  :Very Happy: 

----------

## AgBr

 *AngelKnight wrote:*   

> This wiki specifically says to make the host machine the defaultrouter in the guest, not the actual network's bridge.  This is probably why.
> 
> Trying to use the real network segment's defaultrouter will not work for this configuration as you have discovered.
> 
> This wiki needs quite a bit of fixing around networking actually.  Ugh.  I'm too drunk to fix this now.  Anyone? 

 

I am shy to do it as my knowledge about these things is too limited as you know. But I have made a comment about my experiences in the talk section already with reference to this thread.

----------

