# Alternative network filesystems? (nfs performance)

## bas_s

Hi all,

Ive got a fileserver @home with 4*120GB in software raid-5

the server is connected to my workstation with GigE NIC's over a crosslink.

on the server I get 75MB/s (600Mbit/s) read speeds

over the GigE cable I get:

480 Mbit/s transfers with netperf

290 Mbit/s transfers with ftp

165 Mbit transfers with NFS

105 Mbit transfers with samba

The GigE NIC's are set to MTU=9000

I've tried all sorts of combinations with nfs (nfsv3, nfsv4, tcp, rsize, wsize, (a)sync /proc/sys/net/core/[rw]mem)

165Mbit/s seems the highest achievable speed.

With SMB Ive tried a couple of options I found with google.

If I were to achieve 200/250Mbit/s I'd be happy.

Can anyone enlighten my on coda or afs?

Or some nfs speed tweaks ?

Why ftp is so much faster?

below is output of vmstat 1 of the server during nfs and ftp

with NFS cpu seems quite heavily loaded and there is a lot of waiting for IO

double the interupts and triple context switches of ftp

```

NFS

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

 1  8      0   3164   8108 452628    0    0 20956     0 14298  8879  0 83  0 17

 0  8      0   2904   8024 453300    0    0 21472     0 14573  8987  1 83  0 16

 1  8      0   3544   4420 456824    0    0 21768     0 14775  8654  0 86  0 14

 1  8      0   3224   4424 457216    0    0 21092     4 14402  8766  0 84  0 16

FTP

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

 2  0      0   2820    212 465452    0    0 33756     0 7016  2882  7 72 22  0

 1  0      0   2804    220 465460    0    0 33792     0 6523  2652  4 68 28  0

 0  0      0   2868    220 465428    0    0 36864     0 7106  2947  7 73 21  0

 1  0      0   2296    220 466048    0    0 31324     0 7137  2942  6 72 22  0

```

information:

server:

amd 2400+

512 ram

Intel pro/1000 NIC

18GB scsi for OS

1* LSI 1010-66

2* 3ware 7000-2

4*120GB seagate barracuda 7200.7 data

raidtab with chunksize=1024

formated with xfs sunit=256

kernel 2.6.3

workstation:

dual xeon 2.4Ghz

2GB ram

Intel pro/1000 NIC

kernel 2.6.4

Thanks in advance

----------

## flavio

NFS can provide very high performance if tuned properly. In order to do that you need to understand all the subsystems involved and that they usually require specific tweaking.

In the case of NFS, the components are:

a) Physical storage

b) Filesystem

c) I/O to the storage

d) Caching

e) Network protocol

f) Network stack

g) Ethernet driver

All and each one of these subsystems can be tweaked and you can run some tests utilities and gather statistics for each of those.

In order to optimize your system you need to know what the bottleneck is. In your case, the results from vmstat show a very high system (CPU) load which is likely to be the first reason. High system loads mean that most of your CPU is spent on waiting for I/O to be performed (or dedicated to perform it) and that involves both, disk and network I/O. At the same time you see that 8 processes are blocking almos all the time (waiting for that I/O) and that a high percentage of memory is dedicated to cache (which is good).

So, next step, go ahead and run iostat -x while you're transferring a very large file and find out if you're maxing out your hard drives. If that's the case you may need to enable DMA (if you didn't do so before and you have IDE drives), tune the hard drive parameters, disable atime in your filesystem, tune the disk elevator and/or change the configuration of your Raid 5 to make it a volume set without parity (if you don't need the parity).

Then download iperf (look for it in google) and run it between your server and your client and determine if the network is properly configured. You may want to increase the size of the network buffers, interface txqueuelen and tweak certain TCP parameters (if you decide to go with NFS over TCP).

Finally you need to decide if you want to go with NFS over UDP or TCP (each one has its advantages and disadvantages) and tweak the NFS parameters (increase buffers size and so on).

As a bottom comment and if you need more specific guidelines just do a search on google for nfs tweaking on linux.

----------

## bas_s

 *flavio wrote:*   

> NFS can provide very high performance if tuned properly. In order to do that you need to understand all the subsystems involved and that they usually require specific tweaking.
> 
> In the case of NFS, the components are:
> 
> a) Physical storage
> ...

 

But Ive done all this.

your points a+b+c

read speed on the server is 75MB/s (600mbit/s)  

your point d

The test file is 3GB in size and cant be cached.

your point e

thats the bugger.. isnt it.. netperf and ftp are fast, nfs is slow.. (my question to start with)

your points e & f

negated.. netperf gets near 500mbit/s performance..

Like I said.. 

I tried:

- google. 

- tcp and udp

- rsize and wsize

- nfs v3 and v4

- sync and async

I tried a gazilion different combinations of all above..

----------

## flavio

An step-by-step guide on how to optimize your nfs server and clients would be too long for here, but you'll find a good guide at:

http://nfs.sourceforge.net/nfs-howto/performance.html

It addresses many of the points that I talked about before.  And don't dismiss the use of iperf (not netperf) to measure real throughput and test it also with different TCP windows (useful if you're considering NFS over TCP) or block sizes (useful if you're considering NFS over UDP).

----------

## lblblb

First of all, thanks for doing due dilligence in trying things on your own.

yes, the nfs how-to has optimization settings which will probably help you.

Just out of curiousity, what transfer rates do you get for each of those, using the same options as the numbers you provided before, if you mount the network filesystems over the loopback interface? (127.0.0.1)

Run and nfs, ftp, netperf, smb, etc., server on the lo interface, start the clients (map the drives) from lo, and run the tests locally.

Pass along the numbers.

----------

