# slow nfs performance[fixed by going with iSCSI]

## madchaz

Hello everyone,

I setup a diskless install for my desktop on my server. Basically, my desktop's root sits on nfs on the server and the system is completly net-booted. (PXE boot, etc)

Generally speaking, the system works beautifully. however, when trying to do emerge, the system ends up spending a huge amount of time just "waiting". Lots of disk activity on the server, even if there is almost no bandwidth use, but pretty much no cpu usage anywhere but wait and the network is sleeping. 

Anyone as an idea how I could speed things up a bit? My 6 core Phenom 2 asn't goten 1/3 of X emerged and it's been at it for over 10 hours, lol.

----------

## dE_logics

Any statistics on the bandwidth used during the operation?

Also I've seen that the NFS server in my case also [Debian based] has a lot of I/O activity for just 8 MB/s of transfer over NFS.

----------

## madchaz

Bandwidth use is insignificant. Over a gigabit network, I'm not even seeing 500kBytes/sec

Disk is pretty busy, but the bandwidth use there is in line with network, using gkrellm to look at it. 

I've pushed 30MBytes/sec on an encrypted volume on the server (steady, with a 10GB file) over nfs, cifs and iscsi in the past without any issue. (Encryption done on the server, not client side)

Un-encrypted, like the NFS share is for my root, I've been able to saturate the network link with the server being 80% idle CPU wise.

I've increased the number of nfsd processes on the server from 8 to 20. It helped some, but I've reached the limit of how much that improved performances, and it's still laughable. (numbers above are with the 20 processes)

Other things go well, it's really portage that is having a hard time. Wish iscsi worked with the latest stable kernel.

----------

## madchaz

litle more information on the whole system. 

Network: gigabit

Server: AMD 3700+, with 2x2T SARA drives, in a software raid1 array, allocated using LVM to several uses. 

Desktop: AMD Phenom II. 

The root for the diskless install for the desktop is a 10G block device allocated on the raid aray above, as are all of my cifs/nfs shares. The OS for the server isn't on that array, but on different disks. 

The desktop currently as a Windows7 install I did not want to touch. That's why the diskless install. From win7, I can transfer over cifs to an encrypted share at an average of 30MBytes per seconds. With lots of small files, it goes down to maybe 15/20MBytes per second, over the gigabit network. 

On an un-encrypted share, I will go to about 80-90MBytes/sec and hold between those 2 until the transfer is over. Booting from the gentoo livecd or my ubuntu key, I can transfer at the same speed over nfs. (From the disk, even if it's ntfs)

----------

## madchaz

I think I found my problem. Issue is that /var/tmp being on NFS is very bad. way too many small IO operations. 

Guess I will need to wait for a patch to come out for the iscsi package and move some of the more IO intencive stuff to it, see if it works better.

----------

## Hu

If you have enough memory on the client, you could build in a tmpfs.  That will avoid the network roundtrips for builds without requiring a write to your Windows hard drive.

----------

## madchaz

You suggesting tempfs for /var/tmp?

I don't think I'd have enough ram to provided it with the space it needs. Only got 4gig on my desktop.

Got  open-iscsi installed. Will see if I can't move some of the activity over to a iscsi device and see if that helps.

----------

## dE_logics

Did you try moving /var/tmp out to a local partition?

----------

## Hu

 *madchaz wrote:*   

> You suggesting tempfs for /var/tmp?
> 
> I don't think I'd have enough ram to provided it with the space it needs. Only got 4gig on my desktop.
> 
> Got  open-iscsi installed. Will see if I can't move some of the activity over to a iscsi device and see if that helps.

 I meant only /var/tmp/portage, but yes, you got the general idea.  4GB RAM should let you create a tmpfs that is adequate for small packages, but I would stay away from hogs like Firefox, Thunderbird, and LibreOffice.

----------

## madchaz

 *dE_logics wrote:*   

> Did you try moving /var/tmp out to a local partition?

 

If you re-read the opening, I do not HAVE local partitions. 

In the end, I moved /var/tmp to an iSCSI disk. Performance went back to what I would have expected from the machine during compilation. 

This tells me the nfs server just couldn't handle the ton of IO required during compilation. It also got rid of some weird timestamp issues I was having, even if both systems are on exactly the same time.

----------

## madchaz

 *Hu wrote:*   

>  *madchaz wrote:*   You suggesting tempfs for /var/tmp?
> 
> I don't think I'd have enough ram to provided it with the space it needs. Only got 4gig on my desktop.
> 
> Got  open-iscsi installed. Will see if I can't move some of the activity over to a iscsi device and see if that helps. I meant only /var/tmp/portage, but yes, you got the general idea.  4GB RAM should let you create a tmpfs that is adequate for small packages, but I would stay away from hogs like Firefox, Thunderbird, and LibreOffice.

 

X would have also been a problem, and I did need to compile that. In the long term, not really a practical solution

----------

