# NFS speed over virtual machines/qemu

## eccerr0r

I wonder what people are seeing for NFS transaction speeds to virtual machines on the same host.

Setup: Core2 Quad Q9550S with 6GB RAM (DDR2/PC5300)

QEMU/KVM on host

Server disks is LVM on RAID5.  All partitions except boot on LVM.

NFS server running on host to serve out LVM partitions

Networking:  Software bridge, QEMU network via macvtap/macvlan.  virtual machine using NFS.

Problem: NFS is extremely slow.  For small files I'm seemingly getting IOPS of like 5 or so.  When I get queued SMTP from remote hosts on my virtual machine, the IOPS is so slow that it can timeout when sendmail runs out of queue slots.

Interactive performance of shells is also very slow due to file open/close/lock speeds.  I think after the files get opened/closed the performance is OK.

Any ideas how to speed things up?  Should it be this slow?  What kinds of speeds are you seeing?  Perhaps LVM or RAID5 is what's causing issues?

----------

## szatox

 *Quote:*   

> For small files I'm seemingly getting IOPS of like 5 or so.

  How you measure it so I could repeat it?

If it's getting slower for small files, I'd look at NFS itself first.

/etc/conf.d/nfs, the following line

# Number of servers to be started up by default

OPTS_RPC_NFSD="8"

This is the first thing that might need a big increase.

Another thing, is your IO blocking or parallel on the client side? (Does initializing session have a big impact?)

 *Quote:*   

> starting machine testmachine with command
> 
> qemu-system-x86_64 -name testmachine -enable-kvm -m 2048 -smp cores=2 -net nic,model=rtl8139,macaddr=DE:AD:BE:EF:8F:2E,vlan=1 -net tap,ifname=testmachine-1,script=no,downscript=no,vlan=1
> 
> 

 

Well, it's an ancient image build without virtio, so it's pretty obvious where's the bottleneck, but let's see it anyway.

running "tar -c / | dd of=/dev/null" on qemu with squashfs over nfs root, so unfortunately it's one big file, but still:

iftop at host:

```

TX:             cum:   1.31GB   peak:   76.1Mb                                                                               rates:   74.6Mb  73.2Mb  73.2Mb

RX:                     250MB           14.2Mb                                                                                        13.9Mb  13.6Mb  13.6Mb

TOTAL:                 1.56GB           90.3Mb                                                                                        88.6Mb  86.8Mb  86.8Mb
```

a bit later

```
TX:             cum:   4.31GB   peak:   76.3Mb                                                                               rates:   55.9Mb  40.5Mb  49.5Mb

RX:                     816MB           14.2Mb                                                                                        8.55Mb  6.20Mb  8.08Mb

TOTAL:                 5.11GB           90.6Mb                                                                                        64.5Mb  46.7Mb  57.6Mb

```

and it keeps 2 nfsd bussy:

```
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                  

12588 me        20   0 2761248 2.051g  34536 S 150.7 28.1  24:49.99 qemu-system-x86                                                                          

 4743 root      20   0       0      0      0 S  16.9  0.0   3:05.92 nfsd                                                                                     

 4742 root      20   0       0      0      0 R   9.9  0.0   1:44.86 nfsd                                                                                     

 7357 me        20   0 1369448 474248  92620 S   7.9  6.2 124:05.33 seamonkey                                                                                

 4174 root      20   0  305836  63568  34856 S   5.9  0.8  63:32.69 X                                                                                        
```

On the other hand, running tar against /usr/portage (without distfiles) gives ~4Mbps while running separate tar for every directory inside /usr/portage (skipping distfiles) gives between 15Mb and 40Mb

----------

## eccerr0r

I suppose the perceived 5 IOPS more like a guess - I should look for a better benchmark but it needs to also test flocks.  The impact is very visible at least to processing sendmail requests.  I think it takes at least a minute to process a single mail from sendmail -> procmail -> write to disk, and if another sendmail request comes it, they seem to clash with each other, almost exponentially increasing time taken to complete.  Come on, this is a small text email, shouldn't take a minute to procmail and write to disk... sometimes I get large batches of emails and sendmail seems to livelock - and I stop getting any mail.

I think just one of the nfsd's on the host/VM server is in D wait most of the time, but others are sleeping.  It's clearly waiting on disk resources.

----------

