# NFSv3 really slow? (was: QEMU issue?)

## eccerr0r

Anyone use QEMU (virtmanager) and NFS to the parent machine?

My VM mounts a NFS share on the host that runs that VM.  For some reason that NFS share is excruciatingly slow, seems like I get only a few MB/min trying to rsync to it (trying to backup the VM disk).  Any ideas how to speed it up?

Any better ways to backup virtual machine images?

EDIT:

I think I have ruled this down to NFSv3 with small files.  There's probably some locking issue going on here.

----------

## azieba

A little more information could help. What is the processor usage and the network usage on host and guest machine ?

----------

## eccerr0r

The (Physical) Network utilization is near 0 - after all this should be RAM-to-RAM copy.  In virt-manager it reports very little network activity on the VM.

The guest/virtual machine is mostly in D-wait, otherwise it is idle.  This implies it's a "host" issue.

The host machine... I never looked at, it's a quad core machine and its load average has always been less than 4.

----------

## azieba

Check just to be safe CPU usage on host machine by nfs process during  the transfer (with top or/and  htop ).

----------

## eccerr0r

```
vm:/nfs/host/tmp$ time dd if=/dev/zero of=temp bs=1M count=2K

2048+0 records in

2048+0 records out

2147483648 bytes (2.1 GB) copied, 102.517 s, 20.9 MB/s

real    1m42.620s

user    0m0.007s

sys     0m2.850s

```

Hmm.  During this time, the host remained mostly idle but load average went up implying a lot of disk wait.  NFSd was remained less than 3% of one core.  qemu remains as #1 CPU utilizer but less than 50% of a core.

At 20MB/sec it's bad but not completely horrible (but much worse than when I had this as a physical machine), it's the small files that really kills, and I tend to work with many small files with this virtual machine... Backup over NFS...

----------

## azieba

I'm not certain but still it feels like something with CPU usage or disk waiting for me. Especially with lots of little files. Try running the backup and then watch CPU usage and disk waiting on (use top ) and maybe io operations on (use iotop) . The dd is not showing te real thing since its reading zeros and not from disk, 

If it's IO problem you should notice high usage of CPU by IO operation and lots of waiting:

Look closely at third line (you should see something like below (it's in percents scale) )

%Cpu(s):  4,6 us,  1,0 sy,  0,3 ni, 93,5 id,  0,7 wa,  0,0 hi,  0,0 si,  0,0 st

----------

## eccerr0r

It almost seems like the server is sleeping between calls.

This was hard to grab a shot when nfsd actually showed up in 'top'.    This is on the host machine:

```
top - 14:55:20 up 2 days, 14:04,  6 users,  load average: 1.71, 0.70, 0.34

Tasks: 209 total,   1 running, 208 sleeping,   0 stopped,   0 zombie

%Cpu0  :  3.9 us,  1.0 sy,  0.0 ni, 94.4 id,  0.3 wa,  0.0 hi,  0.3 si,  0.0 st

%Cpu1  :  0.7 us,  0.0 sy,  0.0 ni, 92.6 id,  6.7 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu2  :  3.0 us,  0.7 sy,  0.0 ni, 94.7 id,  1.7 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu3  :  1.0 us,  1.3 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem:   6104356 total,  4355868 used,  1748488 free,   322772 buffers

KiB Swap:  1048572 total,   178092 used,   870480 free,   988044 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU     TIME+ P   SWAP COMMAND

24358 fgu       20   0 1524764 302532  62120 S 3.322  66:14.99 3      0 firefox

 2784 qemu      20   0 2346580 783600   2052 S 2.990 350:18.70 2  28180 qemu-s+

 2373 root      20   0  671152 343504 208660 S 1.329 226:46.79 3   7800 X

 2478 root      20   0  876652   6740   3472 S 0.997  45:01.60 1   1568 libvir+

 3223 fgu       20   0  988680  74680  20260 S 0.997  33:48.43 2      0 python2

 2291 root      20   0       0      0      0 D 0.332   0:23.64 1      0 nfsd

 2764 qemu      20   0 1541456 130536   1676 S 0.332   6:01.87 1  96776 qemu-s+

```

The 2291 nfsd goes away fairly quickly, and another nfsd thread goes into D state.

This is what I did on the client (process 2784) (though I have another virtually idle qemu at 2764):

```

vm:~$ du -s /usr/include/

235548  /usr/include/

vm:~$ time cp -a /usr/include /nfs/host/tmp/test

(aborted after 20 minutes)

```

I just tested this on a physical machine and noticed it's just as slow.  Hmm.  This is a NFS issue and not QEMU.

NFS mount options: rw,nosuid,nodev,hard,intr,sloppy

----------

## azieba

CPU and IO seems fine, hmm. From what I read NFS is always much slower with lots of small files, maybe some optimization may help,

modifying rsize and wsize maybe.

There should be something on NFS tuning here http://nfs.sourceforge.net/nfs-howto/ar01s05.html

----------

## eccerr0r

I wonder if asynchronous writes are enabled default.

I suppose I don't have to worry about the nfs server machine going down in this QEMU case...

----------

