# Confused about KVM Network Configuration

## dustfinger

Hi,

I am completely confused about networking a qemu kvm. I will mention now that I have not yet created the vm. I am trying to sort out the networking so that when I run the command to create the vm I can pass the network details.

Goal: to create a kvm that has its own static IP address.

Current Setup

The host has a single physical network adapter and the interface eth0 configured with a static IP address and a vlan eth0.0 configured with a different static IP address.

/etc/conf.d/net

```

config_eth00="<static-ip-1> netmask 255.255.255.0 brd <static-ip-1-fragment>.255"

routes_enp3s0f0="default via <static-ip-1-fragment>.1"

vlans_eth0="2"

eth0_vlan2_name="vlan2l"

config_vlan2="<static-ip-2> netmask 255.255.255.0 brd <static-ip-2-fragment>.255"

```

I have the second IP address for the kvm. I created vlan2 with the intention that the kvm would use that interface. I have since read that I need to use a bridge. I have read the wiki on networking with qemu as well as as the wiki on network bridging and am thoroughly confused.I think that part of my confusion stems from the fact that the wiki does not explain what it intends to achieve with the particular configuration that they outline. Simply put, I am confused because:

1. Their example does not use a vlan.

2. They set config_eth0=null whereas I am starting off with eth0 set to a static ip and my vlan eth0.2 set to a static ip.

3. They configure the bridge interface br0 to ether dhcp or a default local IP.

My Questions:

1. Why do I really need a bridge?

2. Should I be setting my vlan in the same way that they wiki sets eth0: config_vlan2=null and config_br0="<static-ip-2> netmask 255.255.255.0 brd <static-ip-2-fragment>.255" then bridge_br0="vlan2 tap0"

3. How can I go about configuring /etc/conf.d/net so that I can reach my goals as described above?

Sincerely,

dustfinger

-- EDIT: --

#Here is my command for creating the vm.

#Note that I am creating it on a zvol

#So far this does not work  :Sad: 

```

VNIC=vlan2

HDD=tank/vm/prod-web-1 #My zvol

CD=~/vm/livedvd-amd64-multilib-20140826.iso

mac=`ip link show vlan2 | awk '/ether/ {print $2}'`

qemu-system-x86_64 -enable-kvm \

        -boot cd \

        -global ide-drive.physical_block_size=4096 \

        -drive file=$HDD,if=virtio,index=0,cach=none,format=raw \

        -drive file=$CD,media=cdrom,if=ide,index=2 \

        -netdev tap,id=t0,ifname=$VNIC,script=no,downscript=no

        -device e1000,netdev=t0

        -machine pc,accel=kvm,iommu=on \

        -cpu host \

        -smp 4 \

        -m 8G,slots=2,maxmem=16G \

        -name prod-web-1 \

        -usb \

        -nographic

```

----

----------

## NeddySeagoon

dustfinger,

A bridge is the software equivelent of a router - every packet sent to the bridge goes to every device connected to the bridge.

On the host, you set  

```
config_eth0="null" 
```

so that you can donate the interface to a bridge.

The bridge gets an IP address in the normal way and works on the host as would eth0.

You use the tun/tap driver to export connections from the bridge to your VMs.

Inside your VMs they have a network interface that works in the normal way.

They can use dhcp to get an address from your dhcp server or you can set up the net file with a static address.

Everything connected to the bridge will be in the same subnet as the bare metal host.

The bare metal setup .. all static.

```
config_eth0="null"

config_eth1="null"

config_eth2="null"

config_eth3="null"

config_eth4="null"

# My public subnet

config_br0="xx.yy.zz.ww/29"

# the DMZ

bridge_br1="eth2"

config_br1="192.168.10.254/24"

# wireless

bridge_br2="eth3"

config_br2="192.168.54.254/24"

# protected wired

bridge_br3="eth4"

config_br3="192.168.100.254/24"
```

The bridges are all passed to a VM to do firewalling, where they appear as eth0 ... eth3.

The VM is started as

```
/usr/bin/qemu-system-x86_64 -name Router -S -machine pc-0.14,accel=kvm,usb=off -m 1024 -realtime mlock=off \

-smp 2,sockets=2,cores=1,threads=1 -uuid 19860cc8-b232-1cad-c562-15b18883886a -no-user-config -nodefaults\

-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Router.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control\

-rtc base=utc -no-shutdown -boot menu=off,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \

-drive file=/dev/vm/router,if=none,id=drive-virtio-disk0,format=raw \

-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio\disk0,bootindex=1 \

-netdev tap,fd=20,id=hostnet0,vhost=on,vhostfd=21 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:2b:f1:3e,bus=pci.0,addr=0x6 \

-netdev tap,fd=22,id=hostnet1,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:b7:f5:01,bus=pci.0,addr=0x5 \

-netdev tap,fd=24,id=hostnet2,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:ad:25:ea,bus=pci.0,addr=0x7 \

-netdev tap,fd=26,id=hostnet3,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet3,id=net3,mac=52:54:00:ab:b1:41,bus=pci.0,addr=0x8 \

-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \

-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
```

I didn't write that command line, virt-manager did.

The use of the tap device to connect tho the host bridges can clearly be seen.

Why don't you want to use a bridge?

From the inside, the VM behaves in all respects as another system, even though its a program executing on the host.

Inside the KVM, I have  

```
config_eth0="192.168.10.253/24 brd 192.168.10.255"

config_eth1="192.168.100.253/24 brd 192.168.100.255"

config_eth2="null"

config_eth3="192.168.54.253/24 brd 192.168.54.255"
```

config_eth2="null" is because the router is a PPPoE end point, so no IP address is needed on the interface that provides the PPPoE link.

----------

## dustfinger

Hi NeddySeagoon,

  Thank you so much for your time and advice. In your example you show how you bridge several physical interfaces: eth0, eth1, eth3 etc. I only have one network card and so I have only one physical interface: eth0. I do have 2 static ip addresses though. I would like the host to have one static ip address and the vm to have the other static ip address. In the example below I attempt to follow your pattern, but I create a vlan off of the physical interface for the second IP address. I then try to create two bridges, one for eth0 and one for my vlan (web1). Unfortunately this did not work. The result was that tap0, eth0 and web1 all appeared to start, but did not have the static IP address assigned to them. Trying to start /etc/init.d/net.br0 would just complain that br0 required services web1 and tap0. Trying to start /etc/init.d/net.br1 would just complain that net.br1 failed to start.

```

#So that I can donate the interface to a bridge

config_eth0=null

#so that I can have a vlan for my second IP address

vlans_eth0="2"

eth0_vlan2_name="web1"

#so that I can donate the interface to a bridge

config_web1=null

#Configure TUN/TAP interface

tuntap_tap0="tap"

# tap0 defined empty

config_tap0=null

# Configure network bridge for the physical interface eth0

config_br0="198.xx.xx.225 netmask 255.255.255.0 brd 198.xx.xx.255"

routes_br0="default via 198.xx.xx.1"

mac_br0="aa:bb:cc:dd:ee:ff"

bridge_br0="eth0" # add all interfaces to bridge

rc_net_br0_need="eth0" # we need run eth0 before create bridge!

# Configure network bridge for the vlan interface

config_br1="192.yy.yy.241 netmask 255.255.255.0 brd 192.yy.yy.255"

mac_br1="aa:bb:cc:dd:ee:ff" #

bridge_br1="web1 tap0" # add all interfaces to bridge, usually use one TUN/TAP interface for one Vritual Machine. In this example we have one vm

rc_net_br1_need="web1 tap0" # we need run web1 and tap0 before create bridge!

```

How can I correctly make use of both my static IP addresses with only one physical ethernet?

Sincerely,

dustfinger.

-- EDIT --

Hi,

I decided to take a step back and configure a much simpler senario. So I decided to only create a bridge for the physical interface eth0 as follows:

```

config_eth0=null

#Configure TUN/TAP interface

tuntap_tap0="tap"

# tap0 defined empty to avoid DHCP being run for their configuration

config_tap0=null

# Configure network bridge

config_br0="198.xx.xx.225 netmask 255.255.255.0 brd 198.xx.xx.255"

brctl_br0="setfd 0

sethello 10

stp off"

routes_br0="default via 198.xx.xx.1"

mac_br0="aa:bb:cc:dd:ee:ff"

bridge_br0="eth0" # add all interfaces to bridge, usually use one TUN/TAP interface for one Vritual Machine. In this example we have one VM.

```

Now when I run 

```

$ sudo /etc/init.d/net.br0

* Bringing up interface br0

*   Creating bridge br0

add bridge failed: package not installed

* ERROR: net.br0 failed to start

```

I am not sure what package it is referring to. I already have  net-misc/bridge-utils installed.

----------

----------

## NeddySeagoon

dustfinger,

Its an entry in /etc/init.d/ its looking for.

```
config_br0="198.xx.xx.225 netmask 255.255.255.0 brd 198.xx.xx.255" 
```

is a static setup.

Much as you now have an net.eth0 -> net.lo in /etc/init.d/, you need a net.br0 -> net.lo symlink

You should remove the net.eth0 -> net.lo symlink meanwhile.

You also need to swap entries in your default runlevel if you want br0 to start automatically.

 --- edit ---

You have it

```
$ sudo /etc/init.d/net.br0 
```

Hopefully you had start on the end of that.

Try 

```
$ sudo /etc/init.d/net.br0 -v start
```

The  -v is verbose made.

You need bridge support in your kernel too.

----------

## dustfinger

Hi NeddySeagoon,

I have not been able to respond until now because I have been burdened with other work. I will be back to working on this issue tonight though. I wanted to post an update because I have changed my configuration somewhat and made some progress.

Please see my network diagram which illustrates what my plan is.

The following is my current /etc/conf.d/net configuration:

```
config_eth0=null

vlans_eth0="2 3"

eth0_valn2_name="eth0.2"

config_eth0_2="198.27.xx.25" netmask 255.255.255.0 brd 198.27.xx.255

routes_eth0_2="default via 192.27.xx.1"

eth0_valn3_name="eth0.3"

config_eth0_3="192.95.xx.33" netmask 255.255.255.0 brd 192.95.xx.255

mac_eth0_3="00:50:56:xx:yy:zz" # virtual mac assigned by OVH to 192.95.xx.33

dns_servers="8.8.8.8 8.8.4.4" #google's dns servers
```

I got rid of the /etc/init.d/net.br0 and the vlan links in /etc/init.d/. I have found that the vlans are automatically created in /proc/net/vlan. For the moment I am trying to get the vlan up without worrying about bridging.

When I run /etc/init.d/net eth0.2 and eth0.3 are correctly assigned ip addresses and I can ping them from the host, but I cannot ping them from outside the host, nor can I ping google.com. I am now wondering if making a vlan aware bridge would help. Someone on IRC told me that he believed ovh security will not allow me to use vlans. I am wanting to give it a bit more of a try though because it would be really nice to use vlans.

Note, that earlier I did not have a virtual mac address for my second static ip. I have now configured this with ovh: http://help.ovh.com/DedieMac and assign that interface with the virtual mac address.

Today I am going to be mostly busy with thanks giving dinner, but late tonight I will get back to working on this. Thank you for your responses so far. I really appreciate it. I know very little about networking and so there has been a large learning curve for me to understand what vlans and bridges are really used for and why I need them etc.

Sincerely,

dustfinger

----------

## NeddySeagoon

dustfinger,

OVH - Oh dear.

I left them a while ago for Hetzner. I got better hardware - more of everything for the same cost. The support is much better too.  

I hope you do not use the OVH Gentoo offering.

IPv6 for your KVMs is straight forward.

Is there a reason why you appear to be avoiding virt-manager?

The command line is doing it the hard way.

For firewalling, you can set up a bridge with no real hardware assigned, give it a IP address then forward packets to it with IPtables.

It will bridge the filtered packets.

----------

## merky1

First bridges are not like routers.  They are more like switches.  Basically, they provide a virtual bus for the virtual network adapters to "plug" into.  They allow you to do things like create bridges that have no physical adapters (host only isolated networks) and have bridges connected to a physical adapter that does not use an address.

On my system I have a bridge layout like this :

```
br0             8000.18a905775794       yes             enp2s0f0

br1             8000.18a905775796       yes             enp2s0f1

                                                        vnet0

                                                        vnet1

                                                        vnet2

                                                        vnet3

                                                        vnet4

                                                        vnet5

                                                        vnet6

                                                        vnet7

```

br0 is the primary management network, with an address on my "management" vlan.  br1 is connected to my "virtualization" vlan, which is where all the vnet / virtual machines connect.

----------

## dustfinger

 *NeddySeagoon wrote:*   

> OVH - Oh dear.
> 
> I left them a while ago for Hetzner. I got better hardware - more of everything for the same cost. The support is much better too.  
> 
> I hope you do not use the OVH Gentoo offering.
> ...

 I keep hearing that people are not happy with OVH. I did not use their Gentoo template; the first thing I did was ordered a usb drive, downloaded a live cd and installed Gentoo from scratch. I am going to finish the process of setting this server up with OVH, since I am their right now. I will checkout Hezner though as a future option since you recommend it.

 *NeddySeagoon wrote:*   

> 
> 
> Is there a reason why you appear to be avoiding virt-manager?
> 
> The command line is doing it the hard way.
> ...

 I am trying to expand my understanding. I have been doing a whole lot of man page reading and experimenting while setting up this server.

 *NeeddySeagoon wrote:*   

> For firewalling, you can set up a bridge with no real hardware assigned, give it a IP address then forward packets to it with IPtables.
> 
> It will bridge the filtered packets.

 Perfect, that is what I will do!

merky1: Is vnetx the same thing as vlanx? Or is vnet something slightly different? Is vnetx just the interfaces for each virtual machine? I have not yet created my virtual machine, i was trying to solve these networking problems first.

Sincerely,

dustfinger

-- EDIT --

NeddySeagoon, thank you for the tip about using ipv6. That also lead me to libvirt for managing qemu and OpenVSwitch. I have only been reading up on all of this so far, but I will start actually installing and configuring everything tomorrow night.

----------

----------

## NeddySeagoon

dustfinger,

OVH give you a rescue system.  Its a net booted debian based system I think.

That's how I installed my Gentoo at OVH.

They used to provide a way to boot your install into QEMU, so you you could get the console.

Unfortunately, I got Intel real hardware and QEMU pretended to be AMD, so with everything built -march=native, QEMU failed for other reasons.

The OVH forums are helpful.  Some of the OVH support staff post there but official support is fairly limited, to put it kindly.

My OVH install is still alive and well as a KVM on my Hetzner box.

----------

## dustfinger

Hi NeddySeagoon,

I never got a rescue system with ovh. I did get an IPMI (I have always worried about the security of this) which gives me Keyboard Video Monitor access. I used that in conjunction with a usb drive to install gentoo from a live cd.

I am currently considering one of the following options to move forward with:

1. to use libvirt/virsh to manage my qemu kvms and OpenVSwitch to manage my network

2. To use OpenStack to manage everything. The downside here is that I have only one physical host. It does have 12 cpu and 64 GB of RAM though

I think that option 1 would do well for me, but I if I do get more hosts in the future, I don't imagine that it would be very easy to switch to openstack at that point. Whereas, if I just start with openstack now, it might be overkill, but it might also be easy to expand to additional hosts as I acquire and require them.

Sincerely,

dustfinger

----------

## merky1

 *dustfinger wrote:*   

> 
> 
> merky1: Is vnetx the same thing as vlanx? Or is vnet something slightly different? Is vnetx just the interfaces for each virtual machine? I have not yet created my virtual machine, i was trying to solve these networking problems first.
> 
> 

 

vnet[x] is not the same as vlanx.  It is the virtualized NIC's connection to the hosts bridge.  Basically uses a TUN/TAP interface.

I would highly recommend you stay away from the OpenStack stuffs until you grow beyond 5 hosts.  Libvirt/Virtual Manager will do you well until you decide to expose things externally / for money.

----------

