# Converting KVM VMs from Debian to Gentoo, networkproblems

## capasto

Hi guys,

I'm running a rootserver at http://hetzner.de. The server is currently installed with debian 6, there are several vms on the host. I got a new server with gentoo as base OS and have to convert the vms from the debian maschine to the new gentoo one.

The Hetzner network is pretty strange, I have to create local bridge and route every single ip to this bridge. Here is my working debian /etc/network/interfaces file, maybe you need it for better comprehension:

```
auto lo

iface lo inet loopback

auto  eth0

iface eth0 inet static

  address   188.40.74.5

  netmask   255.255.255.255

  gateway   188.40.74.1

  pointopoint 188.40.74.1

auto virbr1

iface virbr1 inet static

  address 188.40.74.5

  netmask 255.255.255.255

  bridge_stp off

  bridge_fd 0

  pre-up brctl addbr virbr1

  up ip route add 188.40.74.35/32 dev virbr1

  up ip route add 188.40.74.36/32 dev virbr1

  up ip route add 188.40.74.39/32 dev virbr1

  up ip route add 188.40.74.61/32 dev virbr1

  up ip route add 188.40.193.200/32 dev virbr1

  up ip route add 188.40.193.201/32 dev virbr1

  up ip route add 188.40.193.202/32 dev virbr1

  up ip route add 188.40.193.203/32 dev virbr1

  up ip route add 188.40.193.204/32 dev virbr1

  up ip route add 188.40.193.205/32 dev virbr1

  up ip route add 188.40.193.206/32 dev virbr1

  up ip route add 188.40.193.207/32 dev virbr1

```

The outpot from "brctl show" at the debian host:

```
bridge name     bridge id               STP enabled     interfaces

virbr1          8000.fe54000c1ef5       no              vnet0

                                                        vnet1

                                                        vnet2

                                                        vnet3

                                                        vnet4

                                                        vnet5

                                                        vnet6

```

important is: eth0 is not part of the bridge. I wasn't able to find any example configuration for this network setup, in every case is eth0 part of the birdge, but this doesn't work at hetzner.  After reading trough the gentoo wiki and /usr/share/doc/openrc-0.9.8.4/net.example.bz2, I created the following /etc/conf.d/net file (I'm unfamiliar with the gentoo syntax in the /etc/conf.d/net file, maybe you can give me some advise?):

```
v6net1=( "2a01:4f8:160:2ffd" )

modules=( "iproute2" )

config_eth0=(

        "${v6net1}::2/64"

        "176.9.145.241 netmask 255.255.255.255 pointopoint 176.9.145.225"

)

bridge_virbr1=""

brctl_virbr1=( "setfd 0 sethello 0 stp off" )

config_virbr1=(

        "${v6net1}::2/64"

        "176.9.145.241 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "176.9.145.249 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "176.9.145.250 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "176.9.145.251 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "176.9.145.174 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.112 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.113 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.114 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.115 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.116 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.117 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.118 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.119 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.120 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.121 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.122 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.123 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.124 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.125 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.126 netmask 255.255.255.255 pointopoint 176.9.145.225"

        "5.9.132.127 netmask 255.255.255.255 pointopoint 176.9.145.225"

)

routes_eth0=(

        "default via fe80::1 dev eth0"

        "default via 176.9.145.225 dev eth0"

)

dns_servers_eth0=( "213.133.98.98 213.133.99.99 213.133.100.100" )

```

output from ifconfig/brctl show/ip a:

```
host03 ~ # ifconfig

eth0      Link encap:Ethernet  HWaddr 10:bf:48:7e:e4:db

          inet addr:176.9.145.241  Bcast:0.0.0.0  Mask:255.255.255.255

          inet6 addr: fe80::12bf:48ff:fe7e:e4db/64 Scope:Link

          inet6 addr: 2a01:4f8:160:2ffd::2/64 Scope:Global

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:5483 errors:0 dropped:0 overruns:0 frame:0

          TX packets:4504 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:465393 (454.4 KiB)  TX bytes:558410 (545.3 KiB)

lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:27 errors:0 dropped:0 overruns:0 frame:0

          TX packets:27 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:1524 (1.4 KiB)  TX bytes:1524 (1.4 KiB)

host03 ~ # brctl show

bridge name     bridge id               STP enabled     interfaces

host03 ~ # ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 10:bf:48:7e:e4:db brd ff:ff:ff:ff:ff:ff

    inet 176.9.145.241 peer 176.9.145.225/32 scope global eth0

    inet6 2a01:4f8:160:2ffd::2/64 scope global

       valid_lft forever preferred_lft forever

    inet6 fe80::12bf:48ff:fe7e:e4db/64 scope link

       valid_lft forever preferred_lft forever

host03 ~ #

```

It seems like at least the eth0 part is right because the server is reachable now, but ifconfig/ip a doesn't list virbr1. Does anybody of you have any ideas?

----------

## cach0rr0

your 'virbr1' interface is created by this line, if i understand correctly

```

pre-up brctl addbr virbr1

```

making the same under gentoo would be something like

```

bridge_br0="eth0 tap0 tap1 tap2 tap3"

```

in /etc/conf.d/net (adjust accordingly, based on whatever you decide to add to the bridge, what you want to name the bridge, etc)

in other words, the name 'virbr1' is arbitrary. You, the user/admin, get to define what the bridge is named. 

I'll have to wait 'til football is done before i respond in more detail

if a full working example helps

```

bridge_br0="eth0 tap0 tap1 tap2 tap3"

brctl_br0="setfd 0 stp off sethello 0"

rc_net_br0_need="net.tap0 net.tap1 net.tap2 net.tap3"

config_br0="192.168.1.85/24"

routes_br0="default via 192.168.1.1"

dns_domain_br0="mydomain.com"

dns_servers_br0="192.168.1.1"

dns_search_br0="mydomain.com"

config_tap0="null"

tuntap_tap0="tap"

tunctl_tap0="-u meat"

mac_tap0="52:54:00:12:34:56"

config_tap1="null"

tuntap_tap1="tap"

tunctl_tap1="-u meat"

mac_tap1="52:54:00:12:34:57"

config_tap2="null"

tuntap_tap2="tap"

tunctl_tap2="-u meat"

mac_tap2="52:54:00:12:34:58"

config_tap3="null"

tuntap_tap3="tap"

tunctl_tap3="-u meat"

mac_tap3="52:54:00:12:34:59"

config_eth0="null"

```

----------

## capasto

I think your example  is wrong because you bridge every interface from VMs with eth0 from the host. 

The problem is that only a routed setup works well at hetzner. I found some pictures in their wiki 

Routed setup: http://wiki.hetzner.de/index.php/Zusaetzliche_IP-Adressen/en#Routed

bridged: http://wiki.hetzner.de/index.php/Zusaetzliche_IP-Adressen/en#Bridged

----------

## cach0rr0

 *capasto wrote:*   

> I think your example  is wrong because you bridge every interface from VMs with eth0 from the host. 
> 
> The problem is that only a routed setup works well at hetzner. I found some pictures in their wiki 
> 
> Routed setup: http://wiki.hetzner.de/index.php/Zusaetzliche_IP-Adressen/en#Routed
> ...

 

that's simply to be used as an example to show gentoo syntax for creating a bridge via /etc/conf.d/net

You asked:

 *capasto wrote:*   

> 
> 
> but ifconfig/ip a doesn't list virbr1
> 
> 

 

the point is that 'virbr1', 'br0', these are arbitrary names, created by brctl. You could name the interface 'capasto0' if you liked; this is determined by how brctl is invoked. 

The full example I posted is actually *my* setup. I do not expect it to be correct for your setup. 

As to Hetzner - I have done bridging with their EQ4, but have not tried it on their VPS systems.

----------

## capasto

I think I found a half working setup

```
v6net1=( "2a01:4f8:160:2ffd" )

v6net2=( "2a01:4f8:160:23e2" )

#v6net3=""

modules=( "iproute2" )

config_eth0=(

        "${v6net1}::2/64"

        "176.9.145.241 netmask 255.255.255.255 pointopoint 176.9.145.225"

)

bridge_virbr1=""

brctl_virbr1=( "setfd 0 sethello 0 stp off" )

config_virbr1=(

        "${v6net1}::2/64"

        "176.9.145.241 netmask 255.255.255.255 pointopoint 176.9.145.225"

)

routes_eth0=( 

        "default via fe80::1 dev eth0" 

        "default via 176.9.145.225 dev eth0"

)

dns_servers_eth0=( "213.133.98.98 213.133.99.99 213.133.100.100" )

routes_virbr1=(

        "176.9.145.249/32"

        "176.9.145.250/32"

        "176.9.145.251/32"

        "176.9.145.174/32"

        "5.9.132.112/32"

        "5.9.132.113/32"

        "5.9.132.114/32"

        "5.9.132.115/32"

        "5.9.132.116/32"

        "5.9.132.117/32"

        "5.9.132.118/32"

        "5.9.132.119/32"

        "5.9.132.120/32"

        "5.9.132.121/32"

        "5.9.132.122/32"

        "5.9.132.123/32"

        "5.9.132.124/32"

        "5.9.132.125/32"

        "5.9.132.126/32"

        "5.9.132.127/32"

)

```

And why didn't brctl show the virbr1? I forgot to create the /etc/init.d/net.virbr1 file  :Very Happy:  After creating it and setting it up for autostart with "rc-update net.virbr1 boot" the bridge exists now after a reboot.

I can create VMs, libvirt automatically creates the vm network device (vnetX) and adds it to the bridge.  

If I specify a ip adress from virbr1 in the vm, the vm can send packets to the internet and gets responses, but the host doesn't forward the responses from eth0 to virbr1.

So everything from virbr1 gets forwarded to eth0 and then goes into the internet (proofed with tcpdump), but the other way doesn't work.

output from sysctl -p:

```
host03 ~ # sysctl -p

net.ipv4.ip_forward = 1

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

net.ipv6.conf.all.forwarding = 1

net.bridge.bridge-nf-call-arptables = 0

net.bridge.bridge-nf-call-ip6tables = 0

net.bridge.bridge-nf-call-iptables = 0

net.bridge.bridge-nf-filter-vlan-tagged = 0

host03 ~ # 
```

 I don't think that iptables blocks anything:

```
host03 ~ # /etc/init.d/iptables status

 * status: stopped

host03 ~ # iptables -L

Chain INPUT (policy ACCEPT)

target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination         

host03 ~ # 
```

 Any ideas?

----------

## cach0rr0

your "routes_virbr1" does not look correct

can you post output of route -n ?

routes_* should have a 'via' in there somewhere.

EDIT:  these are point to point? I still do not think this is correct. route -n should show us this. I have done GRE tunnels and the configuration is in a much different place.

----------

## capasto

I found this working setup now:

```
v6net1=( "2a01:4f8:160:2ffd" )

v6net2=( "2a01:4f8:160:23e2" )

modules=( "iproute2" )

config_eth0=(

        "${v6net1}::2/64"

        "176.9.145.241 netmask 255.255.255.255 pointopoint 176.9.145.225"

)

routes_eth0=( 

        "default via fe80::1 dev eth0" 

        "default via 176.9.145.225 dev eth0"

)

dns_servers_eth0=( "213.133.98.98 213.133.99.99 213.133.100.100" )

```

Sadly, it fails to create a bridge without any ports bridges in or ip adresses configured. So I wrote this dirty init script(/etc/init.d/net.virbr1):

```
#!/sbin/runscript

# Copyright 1999-2012 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

# $Header: $

#depend() {

#

#}

start() {

ebegin "starting init script for virbr1"

einfo "creating bridge"

brctl addbr virbr1

einfo "setting stp off"

brctl stp virbr1 off

einfo "set fd and hello"

brctl setfd virbr1 0

brctl sethello virbr1 0

einfo "bring bridge up"

ip link set dev virbr1 up

einfo "bringing up several ip adresses after a short sleep"

sleep 2

ip route add 176.9.145.249/32 dev virbr1

ip route add 176.9.145.250/32 dev virbr1

ip route add 176.9.145.251/32 dev virbr1

ip route add 176.9.152.174/32 dev virbr1

ip route add 5.9.132.112/32 dev virbr1

ip route add 5.9.132.113/32 dev virbr1

ip route add 5.9.132.114/32 dev virbr1

ip route add 5.9.132.115/32 dev virbr1

ip route add 5.9.132.116/32 dev virbr1

ip route add 5.9.132.117/32 dev virbr1

ip route add 5.9.132.118/32 dev virbr1

ip route add 5.9.132.119/32 dev virbr1

ip route add 5.9.132.120/32 dev virbr1

ip route add 5.9.132.121/32 dev virbr1

ip route add 5.9.132.122/32 dev virbr1

ip route add 5.9.132.123/32 dev virbr1

ip route add 5.9.132.124/32 dev virbr1

ip route add 5.9.132.125/32 dev virbr1

ip route add 5.9.132.126/32 dev virbr1

ip route add 5.9.132.127/32 dev virbr1

eend $?

}

```

Dirty but works fine

----------

