# Problems with kvm, bridged interface, and nfs server on host

## rich0

I figured I'd try posting this here and see if anyone has any suggestions.

I was happily running an NFS server (NFS v3), supporting various clients including some that use an NFS root.  I also used this box as a host for virtualbox.  The box was running gentoo-sources-3.10.17.

Then I decided to migrate from virtualbox to KVM.  In order to run KVM with bridged networking a bridged interface must be set up.  I set this up using openrc to manage it:

config_eth0="null" 

bridge_brkvm="eth0"

config_brkvm="192.168.0.5/24"

brctl_brkvm="setfd 0

        sethello 30

        stp off"

(I got that recipe off of some kvm site, perhaps Gentoo-related, though it is all a blur now.)

That worked fine for getting VMs to work, but I noticed that my NFS clients were unable to connect to the server.  I'd get a ton of messages on the server:

Dec 20 09:47:36 rich rpc.mountd[4323]: authenticated mount request from 192.168.0.10:1015 for /data/diskless/mythliv2 (/data/di

skless/mythliv2)

(the port number varies, but I'll get one of these every 20 seconds or so as the box tries to mount its root)

I can't connect to NFS from VMs either, but rpcinfo shows all the services running as far as I can tell from either the host or a client (obviously I couldn't test from the boxes that use an NFS root)).  After a lot of hair-ripping I got it to work by enabling NFS v4 on the clients and server - I didn't change anything in my configs otherwise.  

I was content to leave it at that, but I just tested out gentoo-sources and vanilla upstream 3.12.5 and on both of those NFS breaks again, with the same error.  I suspect that the issue is related.  I haven't gotten much further than that on debugging 3.12.5, but I figured I'd toss this out there and see what others think.  There were obviously NFS changes in 3.12.5 as there are a few new config options.  One involves supporting security labels for NFS, and the other involves NFS v4.2 support.  I get the same problem whether I set those both to N or Y.

I didn't try disabling the bridge interface on 3.12.5 yet.  That would be easy enough to do but it obviously won't work for KVM that way.  Since NFS is served by the kernel itself I suspect that somewhere along the way a layer is getting skipped so that the kernel isn't actually listening on the right interface.  Other daemons on both the host and VMs seem unaffected by the interface change.

----------

## szatox

Check out my way to make quemu happy

/etc/conf.d/net snippet

```

# Configure the bridge - "man brctl" for more details

config_vn0="10.0.1.1/24"

brctl_vn0="setfd 0

sethello 10

stp off"

rc_net_vn0_provide="lan !net"

```

qemu starting script ( forwards command line params to qemu, feel free to run it as `./qemu_launcher -hda <some disk image>` )

```
#! /bin/bash

### let's define some variables to make everything neat and nice

BRIDGE=vn0

USER=$(whoami)

IFNAME=qtap$RANDOM

MAC=$( printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256)) )

echo "Trying to run qemu with networking up"

sudo /usr/bin/tunctl -u $USER -t $IFNAME && echo "Interface $IFNAME created"

sudo /sbin/brctl addif $BRIDGE $IFNAME && echo "Interface $IFNAME attached to $BRIDGE"

sudo /bin/ifconfig $IFNAME up promisc  && echo "Interface $IFNAME up"

qemu-kvm -net nic,macaddr="$MAC" -net tap,ifname="$IFNAME",script="no",downscript="no" -m 1024 -smp 3 $@

echo "Cleaning up the mess:"

sudo /bin/ifconfig $IFNAME down && echo "Interface $IFNAME down"

sudo /sbin/brctl delif $BRIDGE $IFNAME && echo "Interface $IFNAME removed from $BRIDGE"

sudo /usr/bin/tunctl -d $IFNAME && echo "Interface $IFNAME destroyed"

```

/etc/sudoers snippet

```
### allow qemu setting vitual interfaces

Cmnd_Alias QEMU = /usr/bin/tunctl, /sbin/brctl, /bin/ifconfig

<my username> ALL=(ALL) NOPASSWD: QEMU

```

obviously you need  module 'tun' to be loaded. 3.10.7 used to load it automagically, with 3.10.17 I had to add boot script for this.

Official qemu way involves running VM as root and then dropping permissions, but I don't like this approach so I've written script above. I'm sure you can tweak it to your needs.

Also in my setup vn0 doesn't enslave any real NICs, I'm running all IP forwarding with iptables rules (so I have auto-configured wan device, and 2 nics + virtual as separate subnets routing to each other and br0 without IP assigned working as internal bridge for qemu network when I'm plaing games with it  :Smile:   Dhcpd is set on this host to be authoritative and bound to 2 lan devices and vn0. Wan device is autoconfigured with ISP's dhcp and br0 is not configured at all as I only want it to mimic a switch the host is not connected to). However, this shouldn't really matter

----------

## rich0

 *szatox wrote:*   

> Check out my way to make quemu happy

 

Have you tested serving NFS from the host with that config?

----------

## szatox

yes, I mounted nfs shares in all directions between host and several VM's. Avahi also works, including mdns. The only thing that didn't work for me was ganglia, but I'm new to it so I wouldn't blame it on virtual network.

----------

## rich0

 *szatox wrote:*   

> yes, I mounted nfs shares in all directions between host and several VM's. Avahi also works, including mdns. The only thing that didn't work for me was ganglia, but I'm new to it so I wouldn't blame it on virtual network.

 

What kernel are you running, and can you grep NFS its config file?

I don't have any issues at all with networking between the guests and host.  The issue is with NFS on the host, and it impacts NFS to non-VMs on the network as well.  I don't see any differences in how the bridge interface is set up.

I'm starting qemu in a different way, but that shouldn't have any impact as the issue exists whether I'm running qemu or not (most of the time the host won't be running qemu anyway, so NFS has to work when qemu isn't running).

----------

## szatox

I've run probably all stable gentoo-sources since 3.7 (right now it's 3.10.17), and a few others

The only thing is NFS config that might couse troubles is that /etc/exports is whitespace sensitive, so all entries must follow this pattern including spaces and lack of those.

```
/usr/portage 10.0.0.0/16(async,rw,no_root_squash)
```

```
CONFIG_NFS_FS=m

CONFIG_NFS_V2=m

CONFIG_NFS_V3=m

# CONFIG_NFS_V3_ACL is not set

# CONFIG_NFS_V4 is not set

# CONFIG_NFS_SWAP is not set

CONFIG_NFSD=m

CONFIG_NFSD_V3=y

# CONFIG_NFSD_V3_ACL is not set

# CONFIG_NFSD_V4 is not set

CONFIG_NFS_COMMON=y

```

But here, if you have problem with making NFS work at all, do you have NFS support in your kernel (particularly server part, as those and client don't depend on each other)? And ofc  net-fs/nfs-utils installed?

Another fail point could be your iptables rules (-P INPUT DROP?). And this one doesn't produce any specific error messages. 

Finaly, NFS4 has changed naming conventions, and configs written for NFS2 and 3 do not work with NFS4 - just in case... Those 2 are in fact most likely

----------

## rich0

 *szatox wrote:*   

> But here, if you have problem with making NFS work at all, do you have NFS support in your kernel (particularly server part, as those and client don't depend on each other)? And ofc  net-fs/nfs-utils installed?

 

Yup - I doubt it would have worked before I changed /etc/conf.d/net if I didn't.  If I disable the bridge and switch back to eth0 it starts working and I don't even need to reboot.  If I switch to NFS v4 it also works on 3.10.17, but not on 3.12.5.

 *Quote:*   

> Another fail point could be your iptables rules (-P INPUT DROP?).

 

I'm not using iptables.  I checked and iptables -L just returns ACCEPT as the default on everything and no rules.

 *Quote:*   

> Finaly, NFS4 has changed naming conventions, and configs written for NFS2 and 3 do not work with NFS4 - just in case... Those 2 are in fact most likely

 

Interesting.  I didn't change my config at all to get it to work when I enabled NFSv4.  I didn't have NFSv4 enabled in my kernel previously.

Various shares are configured differently, and none of them work when things aren't working.  That's why I suspect this is some kind of networking/routing issue and not an NFS issue.

----------

## szatox

Perhaps you have both, NFS 3 and 4 active  (and using 3, or in your particular case it didn't matter) as multiple versions can be supported at the same time. Well, I wasn't interested in v4 enough to dig in more.

Anyway, recently someone reported problems with dhcp on similar setup, where the onlly difference against my setup seemed to be phisical NIC bound/unbound to a bridge. I know it sounds stupid.

So, from the working configuration leave eth0 as it is, create bridge manually with brctl and only bind tap to it and see what happens. (or even assign IP to tap device without bridging it) and see if it works.

Also, have you tried setting sniffer on bridge or tap? Maybe that would hint what's wrong.

----------

