# kvm virtual networking config

## Wizumwalt

I'm using KVM to run a few guest OS's.

My problem is that my guest OS can ping my host OS and get out on the internet, but the host OS can't ping the guest OS.

This is the script I use to setup networking on my host machine.

```

   /sbin/modprobe kvm

   /sbin/modprobe kvm-intel

   /sbin/modprobe tun

   /sbin/brctl addbr br0

   /sbin/ifconfig br0 192.168.100.254 netmask 255.255.255.0 up

   for ((i=0; i < NUM_OF_DEVICES; i++)); do

       /usr/bin/tunctl -b -u $USERID -t qtap$i >/dev/null

       /sbin/brctl addif br0 qtap$i

       /sbin/ifconfig qtap$i up 0.0.0.0 promisc

   done

   echo "1" > /proc/sys/net/ipv4/ip_forward

   iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

```

Here's my /etc/conf.d/net ...

```

bridge_br0="tap0 tap1"

brctl_br0="setfd 0 sethello 0 stp off"

rc_need_br0="net.tap0 net.tap1"

config_eth0="192.168.0.115 netmask 255.255.255.0 brd 192.168.0.255"

routes_eth0="default gw 192.168.0.1"

config_br0="192.168.0.115/24"

config_tap0="null"

tuntap_tap0="tap"

tunctl_tap0="-u name"

mac_tap0="52:54:00:12:34:56"

config_tap1="null"

tuntap_tap1="tap"

tunctl_tap1="-u name"

mac_tap1="52:54:00:12:34:57"

```

Any help much appreciated.

----------

## cach0rr0

```

echo "1" > /proc/sys/net/ipv4/ip_forward

```

you do know you can just set this in /etc/sysctl.conf yeah?

far as the host=>guest (and vice-versa) connectivity, are you specifying a mac address on the kvm command-line? 

for example, for one of my guests (I don't have the problem you do - I did to start, changing the mac and enabling ip_forward fixed it), from the host OS I start up as such:

```

qemu-kvm -drive file=/kvm/web/apache.img,if=virtio,boot=on -net nic,model=virtio,macaddr=00:1d:92:ab:3f:77 -net tap,ifname=tap0,script=no,downscript=no -m 1024 -vnc 127.0.0.1:5 -balloon virtio &

```

Its tap interface on the host OS, however:

```

$ sudo ifconfig tap0

tap0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  

          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:4336312 errors:0 dropped:0 overruns:0 frame:0

          TX packets:5596926 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:500 

          RX bytes:5123382209 (4.7 GiB)  TX bytes:4094951818 (3.8 GiB)

```

my /etc/conf.d/net

```

bridge_br0="eth0 tap0 tap1 tap2 tap3"

brctl_br0=( "setfd 0" "sethello 0" "stp off" )

#RC_NEED_br0="net.tap0 net.tap1 net.tap2 net.tap3"

rc_need_br0="net.tap0 net.tap1 net.tap2 net.tap3"

config_br0=( "192.168.1.85/24" )

routes_br0=( "default via 192.168.1.1" )

dns_domain_br0="whitehathouston.com"

dns_servers_br0="192.168.1.1"

dns_search_br0="whitehathouston.com"

config_tap0=( "null" )

tuntap_tap0="tap"

tunctl_tap0="-u meat"

mac_tap0="52:54:00:12:34:56"

config_tap1=( "null" )

tuntap_tap1="tap"

tunctl_tap1="-u meat"

mac_tap1="52:54:00:12:34:57"

config_tap2=( "null" )

tuntap_tap2="tap"

tunctl_tap2="-u meat"

mac_tap2="52:54:00:12:34:58"

config_tap3=( "null" )

tuntap_tap3="tap"

tunctl_tap3="-u meat"

mac_tap3="52:54:00:12:34:59"

config_eth0=( "null" )

```

note the difference for me with config_eth0

I also notice you don't add eth0 to your bridge. This could be wrong (?)

if it's not the mac address thing, or the non-null eth0 config, I'd be willing to bet if you check dmesg on the host OS you'll see loads of complaints about b0rked ARP requests.

----------

## Hu

What is the command line you use for one of the affected guests?  You have a weird mixture of bridging and NAT.  Your conf.d/net configures one set of bridges, but then your custom script configures a different set.  Your bridge device and your Ethernet card are then given the same IP address.

Do you need to have your guests bridged together?  Your setup would be easier to maintain if you use either a bridge or a NAT, but not both.

----------

## Wizumwalt

Yes, I'm trying to get my guest OS's which are each on separate host OS's talking to each other, as well as be able to access the rest of the network. Both scripts on each machine are almost identical. I start them as such ...

host-1 starts guest like this ...

```

kvm -net nic,macaddr=52:54:00:12:34:56,model=virtio -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-${KVERS} -append "root=/dev/vda console=ttyS0 video=nofb" -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -m 512 -drive file=./gentoo-x86_64.img,if=virtio,cache=none -serial stdio

```

host-2 starts guest like this ...

```

kvm -hda gentoo-copy.qemu -net nic,macaddr=52:54:00:12:34:58 -net tap,ifname=qtap2,script=no,downscript=no -m 512

```

I wasn't sure about where this all belongs. It sounds like ya'll are saying it's either in /etc/conf.d/net, or in my script. One or the other, not both?

----------

## cach0rr0

```

mac_tap0="52:54:00:12:34:56" 

```

 *Quote:*   

> 
> 
> kvm -net nic,macaddr=52:54:00:12:34:56,model=virtio -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-${KVERS} -append "root=/dev/vda console=ttyS0 video=nofb" -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -m 512 -drive file=./gentoo-x86_64.img,if=virtio,cache=none -serial stdio 
> 
> 

 

this is a problem. The mac inside the guest should not be the same as the mac of the tap interface 

I'm pretty sure I ran into that same pitfall. 

by the by, your /etc/conf.d/net says to make the tap device as 'tap0', but your kvm start line says 'qtap0' - i presume this is, as was the case for me, a result of trying to mix multiple sources of documentation that each had their own desired naming convention for this stuff  :Laughing: 

----------

## cach0rr0

what i would do is this:

-ditch that external script

-enable ip forwarding in /etc/sysctl.conf

-make your /etc/conf.d/net look like so:

```

bridge_br0="eth0 tap0 tap1" 

brctl_br0="setfd 0 sethello 0 stp off" 

rc_need_br0="net.tap0 net.tap1" 

config_br0="192.168.0.115 netmask 255.255.255.0 brd 192.168.0.255" 

routes_br0="default gw 192.168.0.1" 

config_tap0="null" 

tuntap_tap0="tap" 

tunctl_tap0="-u name" 

mac_tap0="52:54:00:12:34:56" 

config_tap1="null" 

tuntap_tap1="tap" 

tunctl_tap1="-u name" 

mac_tap1="52:54:00:12:34:57"

config_eth0=( "null" )

```

-reboot host machine

-start your kvm guest with (notice the changed MAC)

```

kvm -net nic,macaddr=00:1d:92:ab:3f:77,model=virtio -net tap,ifname=qtap0,script=no,downscript=no -kernel ./kernel-${KVERS} -append "root=/dev/vda console=ttyS0 video=nofb" -initrd /opt/gentoo-x86_64-initramfs.cpio.gz -m 512 -drive file=./gentoo-x86_64.img,if=virtio,cache=none -serial stdio 

```

after you've started your kvm guest in this way, the NIC will probably be renamed to eth1, since the MAC has changed. 

To fix this, again still inside the guest:

```

rm /etc/udev/rules.d/70-persistent-net.rules

reboot

```

and it should be sorted.

the idea is that not just tap0, tap1, etc, but also eth0, are all added to the bridge, and this is how they communicate with each other

traffic from  your host machine goes out via br0

as does traffic from all your guests as well

and since theyre on the bridge together (with non-conflicting MAC addresses) they can talk to each other just fine. Really that's the crux of the matter, since eth0 isnt part of the bridge, the guests cant talk to it. And since you have the same conflicting mac for both your tap interfaces *and* for their respective guests *inside* the guest OS, routing is a bit fucked

----------

## Wizumwalt

I've still got something flaky. My host OS can ping the bridge, but I can't ping other remote nodes on the network. If I shut the bridge interface down, then I can comm to remote hosts on the network as normal. Heres my current configuration.

/etc/conf.d/net on host OS.

```

bridge_br0="eth0 tap0 tap1"

brctl_br0="setfd 0 sethello 0 stp off"

rc_need_br0="net.tap0 net.tap1"

config_br0="192.168.0.15 netmask 255.255.255.0 brd 192.168.0.255"

routes_br0="default via 192.168.0.1"

config_tap0="null"

tuntap_tap0="tap"

tunctl_tap0="-u user"

mac_tap0="52:54:00:12:34:90"

config_tap1="null"

tuntap_tap1="tap"

tunctl_tap1="-u user"

mac_tap1="52:54:00:12:34:91"

config_eth0="192.168.0.11 netmask 255.255.255.0 brd 192.168.0.255"

routes_eth0="default via 192.168.0.1"

config_eth1="192.168.0.12 netmask 255.255.255.0 brd 192.168.0.255"

routes_eth1="default via 192.168.0.1"

```

I've noticed the mac addr for both eth0 and br0 are the same. Is this correct?

$ ifconfig

```

br0       Link encap:Ethernet  HWaddr 00:1a:a0:a1:aa:97

          inet addr:192.168.0.15  Bcast:192.168.0.255  Mask:255.255.255.0

          inet6 addr: fe80::21a:a0ff:fea1:b487/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:32 errors:0 dropped:0 overruns:0 frame:0

          TX packets:51 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:7538 (7.3 KiB)  TX bytes:9493 (9.2 KiB)

eth0      Link encap:Ethernet  HWaddr 00:1a:a0:a1:aa:97

          inet addr:192.168.0.11  Bcast:192.168.0.255  Mask:255.255.255.0

          inet6 addr: fe80::21a:a0ff:fea1:b487/64 Scope:Link

          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:123 errors:0 dropped:0 overruns:0 frame:0

          TX packets:186 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:20656 (20.1 KiB)  TX bytes:26355 (25.7 KiB)

          Interrupt:16

eth1      Link encap:Ethernet  HWaddr 00:1b:3b:c1:ff:bb

          inet addr:192.168.0.12  Bcast:192.168.0.255  Mask:255.255.255.0

          UP BROADCAST MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

          Interrupt:18 Base address:0x2700

lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:19 errors:0 dropped:0 overruns:0 frame:0 

          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:1960 (1.9 KiB)  TX bytes:1960 (1.9 KiB)

tap0      Link encap:Ethernet  HWaddr 52:54:00:12:34:60

          UP BROADCAST PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:500

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

tap1      Link encap:Ethernet  HWaddr 52:54:00:12:34:61

          UP BROADCAST PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:500

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

```

```

$ sudo sysctl -a | grep bridge

net.bridge.bridge-nf-call-arptables = 0

net.bridge.bridge-nf-call-iptables = 0

net.bridge.bridge-nf-call-ip6tables = 0

net.bridge.bridge-nf-filter-vlan-tagged = 0

net.bridge.bridge-nf-filter-pppoe-tagged = 0

```

/etc/sysctl.conf

```

net.ipv4.ip_forward = 1

```

```

$ brctl show

bridge name     bridge id               STP enabled     interfaces

br0             8000.001aa0a1aa97       no              eth0

                                                                       tap0

                                                                       tap1

```

```

$ brctl showmacs br0

port no mac addr                is local?       ageing timer

1     00:08:74:41:36:1a       no                14.30

1     00:09:5b:c0:b3:dd       no                 3.81

1     00:13:72:1c:ef:c8       no                14.62

1     00:13:72:1d:3e:29       no                14.40

1     00:1a:a0:a1:b4:87       yes                0.00

1     00:1e:ec:19:e2:c6       no                14.95

1     00:25:64:b8:c5:93       no                14.62

2     52:54:00:12:34:60       yes                0.00

3     52:54:00:12:34:61       yes                0.00

```

Last edited by Wizumwalt on Sat Sep 24, 2011 1:23 am; edited 1 time in total

----------

## cach0rr0

your config_eth0 needs to be null. it will do all of its connecting/communication via the bridge, as will your kvm guests

so, this

```

bridge_br0="eth0 tap0 tap1" 

brctl_br0="setfd 0 sethello 0 stp off" 

rc_need_br0="net.tap0 net.tap1" 

config_br0="192.168.0.11 netmask 255.255.255.0 brd 192.168.0.255" 

routes_br0="default via 192.168.0.1" 

config_tap0="null" 

tuntap_tap0="tap" 

tunctl_tap0="-u user" 

mac_tap0="52:54:00:12:34:60" 

config_tap1="null" 

tuntap_tap1="tap" 

tunctl_tap1="-u user" 

mac_tap1="52:54:00:12:34:61" 

config_eth0=( "null" )

config_eth1="192.168.0.12 netmask 255.255.255.0 brd 192.168.0.255" 

routes_eth1="default via 192.168.0.1" 

```

only lines changed in the above are config_eth0, config_br0, and removal of routes_eth0

and yes it your br0 and eth0 mac should be the same

the only stipulation is that you a)dont start two kvm guests with the same mac, b)dont try to have two tap interfaces with the same mac, c)dont start kvm using a mac address that's the same as anything else, inclusive of your tap interfaces (e.g. mac for tap0 should be different from the mac you specify when starting the kvm guest that uses tap0)

----------

## Wizumwalt

Thanks for all your help. It's all making sense to me now.

----------

