# Networked file system with quotas

## grosnours

Hi,

I'm building a small LAMP "cluster" (with file/db replication, failover) and I'm stuck with the filesystem quotas.

My setup is:

- 1 NAS for data storage (files as well as databases), 

- 2+ mysql servers (with replication setup),

- and 2+ apache servers to serve the websites (+ for file access).

The NAS is the only box with harddrives, the others boxes boot on PXE and then with nfsroot.

While the file replication part is not a hard problem, the quotas part is a big one.

What I want to do is to restrict the disk usage of each websites. I tried a ftpd (like pureftpd) and make use of virtual quotas but then, I would have had problems with "nested users" and their quotas and I wouldn't be able to use "real" users. That's why I came to filesystems quotas.

Though it really does what I want while I'm on a local filesystem (restrict disk usage per-uid on a whole mountpoint) and gives error when hardlimit is reached, it does not go so well when it comes to network.

The quotas are applied but NFS does not give any error if hard limit is exceeded. It's confusing for my users because the ftpd does the same while it runs on the NFS share (uploads are ok though file is truncated because hard limit is exceeded).

I'll try SAMBA tomorrow but I'm not very confident even though there's a quota USE flag.

I searched the forum and found some topics about NFS quotas and rpc.rquotad , but it doesn't help.

I need some help here, isn't there any networked filesystem that can handle correctly quotas (ie: give i/o error when quota is exceeded, etc) ?

----------

## gerdesj

It all depends on what your NAS supports.  To be honest it probably runs Samba itself anyway, so that would be the way to go. Check the smbcquotas command out.

----------

## grosnours

Well, the NAS is "just" another box with hardware RAID and multiple gigabit ethernet running Gentoo, so I can run pretty much everything on it.

Thanks for your answer, I'm currently investigating it.

----------

## qubix

How do you spread user information between the 3 boxes? NIS, openLDAP, or standard files (shadow/passwd)?

Why did nfs quotas not work?

You could also try RedHat's cluster file systems aproach. GFS served on a GNBD from the "dev server" to the other boxes might do the trick. Warning - do not try GFS2, as it's not really stable (or was 2-3 months ago).

----------

## gerdesj

 *grosnours wrote:*   

> Well, the NAS is "just" another box with hardware RAID and multiple gigabit ethernet running Gentoo, so I can run pretty much everything on it.
> 
> Thanks for your answer, I'm currently investigating it.

 

Another possibility is EVMS.  You create each area as a separate filesystem and then use EVMS to manage them.  That approach will be unwieldy if you many to manage.  On the other hand there's no way around those "quotas" and its pretty obvious when they get hit!

Cheers

Jon

----------

## grosnours

 *qubix wrote:*   

> How do you spread user information between the 3 boxes? NIS, openLDAP, or standard files (shadow/passwd)?
> 
> Why did nfs quotas not work?

 

I'm currently using openLDAP with nssldap and pam_ldap plugins.

NFS quotas *did* work, the disk usage is limited correctly but there is not any error when you try to exceed that limit (on ne NFS clients).

Let's say a user can have 10MB of disk max. If he's on the NAS (so no NFS) and try to write 11MB of data, he'll get an I/O error or something like that. If he's on another box with the NFS mounted filesystem and write 11MB, he'll get nothing but "ok". It seems NFS does the data transfer but the NFS server simply drops it.

 *Quote:*   

> You could also try RedHat's cluster file systems aproach. GFS served on a GNBD from the "dev server" to the other boxes might do the trick. Warning - do not try GFS2, as it's not really stable (or was 2-3 months ago).

 

Last time I tried GFS, it was rather complicated to setup, but I'll look at it next. Thanks.

----------

## qubix

 *grosnours wrote:*   

>  *qubix wrote:*   You could also try RedHat's cluster file systems aproach. GFS served on a GNBD from the "dev server" to the other boxes might do the trick. Warning - do not try GFS2, as it's not really stable (or was 2-3 months ago). 
> 
> Last time I tried GFS, it was rather complicated to setup, but I'll look at it next. Thanks.

 

Frankly speaking, i suppose you could get more reliability from one server (the NAS server in your setup) with single MySQL+Apache, than 3 servers with GFS+RH ClusterSuite. If your NAS goes down, everything goes down anyway. Just my opinion.

If speed is the case, it might be really worth looking if you really thought it over.

----------

## grosnours

This is only a test setup. I intend to run the files on two boxes with drbd+heartbeat and share that fault-tolerant device to the other boxes. The minor problem with that setup is the big loss of disk space, but that's not the problem to focus for now  :Wink: 

----------

## qubix

Do you really need to have muti-rw access filesystems in your setup? If not, than using DRBD+whatever FS might just give you quotas. If you really need to have multi-rw access to filesystems, than your left with GFS+RH Clusters, or OracleFS if you want to run OracleDB....

----------

## grosnours

Yes, each webserver needs RW access to the same data share and with quotas support. I can't forbid my users to write files in their websites.

As a last resort option, if nothing else works, I can drop filesystem quotas and write a custom script to check, warn and enforce quotas if necessary, but I'd like not to (as filesystems quotas exist for that reason).

----------

## qubix

if that is the case, i would go for RH Cluster with GNBD and GFS for the application and user space, nfsroot/PXE for booting the machines. But:

* that setup is quite unpredictable to live with,

* you have very little control on what is going on with it,

* you cannot export a GNBD and use it on the same system, so your NAS server will only be able to do NAS,

* trying to run apps on only one cluster unit (out of two), when the second is in repair for example, is difficult, especially when you have your data on GFS.

Good luck  :Smile: 

I've tested lots of linux cluster solutions in my previous job, and the outcome is that it's best to have the simplest possible solutions, good backup and application-level replication/clustering if possible.

----------

## mli

Anyone know the solution to the original problem? I have a similar setup with NFS and the user exceeding his/her quota on client does not get any error message and the file is truncated. Quotas are working fine on the server. This situation is very confusing for users because they do not know when they are out of quota (for example when uploading with FTP) and files get corrupted.

----------

