# [Discussion] Network Filesystems

## KermitTheFragger

Hi,

I would like to hear the opinion of the gentoo community (yeah that means you  :Smile:  ) about the network filesystems currently available. And I know you guys have an opinion on this issue  :Very Happy:  . In my eyes NFSv3 is too insecure to be used in a corporate environment (if Im being terribly ignorant, you may say so as well  :Smile:  ) so im looking for alternatives. 

Il start my self (seems only fair):

NFSv3:

Solid, Simple, but insecure, wouldnt want to use it anywhere except for home use. Takes only one smart pants with a knoppix cd to mess things up.

AFS

http://openafs.org/

Great features (client side catching, Kerberos support, etc), Solid, but HUGE to setup and maintain. Seems only usable when you have 100 > users.

NFSv4

http://www.citi.umich.edu/projects/nfsv4/ and http://www.nfsv4.org/ and http://nfsv4.bullopensource.org/

Great features and security (Kerberos), but currently still unstable ??? This seems like a good alternative to NFSv3 (well duh  ) and AFS. But im not sure its production stable (I would really, really like this one to be stable).  

Codafs

http://www.coda.cs.cmu.edu/

interesting features (like being disconnected from the fileserver, and sync later). But also not stable to their own saying, although they having been saying this for years

Intermezzo

http://www.inter-mezzo.org/

Looks nice, but seems dead, or is it me?

CIFS

http://linux-cifs.samba.org/

Beter known as the SMB protocol. Stable. Seems like a viable alternative to NFSv3.

----------

## adsmith

I'm so glad you started this thread.    

I've been using old-school NFS for a long time behind my home firewall, but it's barely a good enough tool.  Like you said, it's certainly too insecure for anything but a home environment, and it lacks all but the most basic features.  

I know my univeristy runs most everything off of AFS, but it doesn't seem like the friendliest system to maintain.  I have no admin experience with it, though.

In my case, all the user info is on my gentoo machine, and my wife's two slow machines (running Fedora and RedHat until we get time to upgrade) essentially act as thick terminals, accessing all data from my machine.  

However, enjoying gentoo so much, I like to play, and playing breaks things, and I have to reboot after a kernel panic, or whatever.  Then my wife gets unhappy, because suddenly her computer freezes as she's doing work -- afterall, all that data is really on my machine!    (Yes, our data is backed up all over the place...)

Anyway, I'd like to play with Coda, specifically for its caching support, but it seems like too much of a pain in the arse to set up at this point.

----------

## KermitTheFragger

Coda looks nice indeed. But im concerd about the stablity. The FAQ says this:

 *Quote:*   

> 
> 
> According to the previous entry from 1998, Coda wasn't ready for production use. What is the current status?
> 
> I'd say a small userbase (20-30 users) and a few servers are pretty workable. Such a setup has been running here at CMU for the past couple of years without significant disasters. 
> ...

 

I dont know from what time period this update in the faq is, but the faq seems pretty out dated. Maybe there is someone on this forum who knows more about this? Because coda looks really nice, but i need something thats production stable.

On the mailinglist is quite some heavy traffic, so the project seems to be alive and kicking.

----------

## brenden

I'm currently looking at OpenAFS.  Big problem is the lack of kernel 2.6 support though.  Is Coda ready for production yet?  I'd really like a high-quality distributed filesystem, rather then using nfs and rsync.

----------

## gvs

I've asked this question in my thread about desktop Linux here a while back:

https://forums.gentoo.org/viewtopic.php?t=230131&highlight=

Coda seems nice, but requires an extra authentication database, AFAIK it can not integrate with kerberos, which AFS can.

AFS is cool, and supposedly stable, but is indeed difficult to set up, and has some quirks. You can run into problems with long running jobs (like downloads) that exceed the lifetime of the kerberos ticket (access will be denied).

nfsv4 seems the most likely candidate in the near future, but is hard to say in what stage the development is, the offcial website is not very clear.

I don't know if it will suffer the same quirks as AFS.

SMB/CIFS is still there, but then again, authentication is a new problem. Do you need to get a samba password? If you mount it, you provide a password on the commandline/file...

In the meantime, Linux is left without an easy and secure network filesystem... (I'm not saying the competition does better, I'm only interested in Linux anyway).

----------

## KermitTheFragger

I posted some questions on the NFSv4 mailinglist, I asked them if it was stable enough for a user base of 15. They told me it is.

Real shame coda doesnt integrate with kerberos. So in my eyes, NFSv4 is the best candidate.

brenden:

I've setup openAFS with MIT kerberos in the past, let me know if you bump into any trouble, maybe i can help you out. Big problem with AFS is in my opinion: You need the krb524 deamon (running a kerberos 4 server). Also it is complex to maintain. Good for a user base of a 1000 user toughs, but probably not a good idea for 15 people with no big needs besides getting access to their files.

----------

## KermitTheFragger

Well I got some updates from the front  :Very Happy: 

I recently found out that samba also had a PAM module:

http://us4.samba.org/samba/docs/man/Samba-HOWTO-Collection/pam.html

Which possibly makes it the most interesting candidate to be NFSv3s successor (for a small user base). Samba has a solid code base, is stable and works well with the 2.6 kernel.

----------

## gvs

 *KermitTheFragger wrote:*   

> Well I got some updates from the front 
> 
> I recently found out that samba also had a PAM module:
> 
> http://us4.samba.org/samba/docs/man/Samba-HOWTO-Collection/pam.html
> ...

 

AFAIK this module is for authenticating against a Samba (or Windows) server.

It does not provide a way to use a CIFS share in Unix without supplying a password at mount time.

----------

## tecknojunky

 *KermitTheFragger wrote:*   

> NFSv3:
> 
> Solid, Simple, but insecure, wouldnt want to use it anywhere except for home use. Takes only one smart pants with a knoppix cd to mess things up.

 So far, it's the only one that has suited my needs.  It's relatively simple and fast.  The drawback... if it's used for /home and it goes down, everyone get frozen.

 *KermitTheFragger wrote:*   

> AFS
> 
> http://openafs.org/
> 
> Great features (client side catching, Kerberos support, etc), Solid, but HUGE to setup and maintain. Seems only usable when you have 100 > users.

 I would have tried it since there's a nice Gentoo howto, but there's no support for 2.6 kernels.

 *KermitTheFragger wrote:*   

> NFSv4
> 
> http://www.citi.umich.edu/projects/nfsv4/ and http://www.nfsv4.org/ and http://nfsv4.bullopensource.org/
> 
> Great features and security (Kerberos), but currently still unstable ??? This seems like a good alternative to NFSv3 (well duh  ) and AFS. But im not sure its production stable (I would really, really like this one to be stable).  

 It's newly installed on my system (hence, why I found this thread).  I can't wait to see if I can take advantage of the replication, migration, load balancing, and much more other features.

 *KermitTheFragger wrote:*   

> Codafs
> 
> http://www.coda.cs.cmu.edu/
> 
> interesting features (like being disconnected from the fileserver, and sync later). But also not stable to their own saying, although they having been saying this for years

 It is also installed on my system, but I simply can't make it play nice.  Whenever I ls in a coda filesystem, nothing is shown.  I got tired of hammering my head on this wall of brick.  Further, whenever you try to search with the term coda, you get a bunch of hits written in italian.  btw, there is a kerboros use flag in the coda ebeuild, so it must integrate with it, no?

 *KermitTheFragger wrote:*   

> Intermezzo
> 
> http://www.inter-mezzo.org/
> 
> Looks nice, but seems dead, or is it me?

 Seem to be quite dead.  There's Lustre that is suppose to take its place, but it seem that the development is stalled.

 *KermitTheFragger wrote:*   

> CIFS
> 
> http://linux-cifs.samba.org/
> 
> Beter known as the SMB protocol. Stable. Seems like a viable alternative to NFSv3.

 I'd like to be able to make samba authenticate Windows users (PDC I think it's called.  Or is it ADS?  Oh well, I'm so not Windows anymore... woohoo!  :Very Happy:  ).  Is this samba-tng that does that?  I know it's a mix of ldap and kerboros, not much more.

----------

## depontius

 *tecknojunky wrote:*   

>  *KermitTheFragger wrote:*   NFSv3:
> 
> Solid, Simple, but insecure, wouldnt want to use it anywhere except for home use. Takes only one smart pants with a knoppix cd to mess things up. So far, it's the only one that has suited my needs.  It's relatively simple and fast.  The drawback... if it's used for /home and it goes down, everyone get frozen.

 

Scares me from a security standpoint. I've seen a little deployment at work, and a decade ago I configured the OS/2 NFS client, including setting UID and GID in a file. THAT'S what scared me to begin, and my impression never got much better. Even though I'd just be running my home network, I still try to use best practices. Right now I'm in the process of bringing up the LDAP/Kerberos/SASL soup, wading through generating certificates that OpenLDAP will be happy with.

 *tecknojunky wrote:*   

>  *KermitTheFragger wrote:*   AFS
> 
> http://openafs.org/
> 
> Great features (client side catching, Kerberos support, etc), Solid, but HUGE to setup and maintain. Seems only usable when you have 100 > users. I would have tried it since there's a nice Gentoo howto, but there's no support for 2.6 kernels.

 

I've used afs at work for over a decade. Works well for single-system-image, but does have some issues. Unix permissions don't really work, especially "group" and "other", but are taken over by the acls. I understand that from a programming standpoint afs doesn't really follow Posix semantics, though I haven't run into this personally. My stuff hasn't been as deep into it as my friends'. There is marginal support for 2.6 now with OpenAFS 1.3, and the whole PAG issue is being resolved, as well. Linus insisted that the PAG issues be resolved in a general way, not the afs-specific way of the original patches. This is going to benefit nfsv4, too.

 *tecknojunky wrote:*   

>  *KermitTheFragger wrote:*   NFSv4
> 
> http://www.citi.umich.edu/projects/nfsv4/ and http://www.nfsv4.org/ and http://nfsv4.bullopensource.org/
> 
> Great features and security (Kerberos), but currently still unstable ??? This seems like a good alternative to NFSv3 (well duh  ) and AFS. But im not sure its production stable (I would really, really like this one to be stable).   It's newly installed on my system (hence, why I found this thread).  I can't wait to see if I can take advantage of the replication, migration, load balancing, and much more other features.

 

I'm holding out for it. Search the forums on nfsv4 and you'll see my name, asking. I also saw your name, apparently successful. I was sorry to see your "nfsv4 overwhelmed" post. I've got Bug 70732 on another window, and will read it right after posting this.

 *tecknojunky wrote:*   

>  *KermitTheFragger wrote:*   Codafs
> 
> http://www.coda.cs.cmu.edu/
> 
> interesting features (like being disconnected from the fileserver, and sync later). But also not stable to their own saying, although they having been saying this for years It is also installed on my system, but I simply can't make it play nice.  Whenever I ls in a coda filesystem, nothing is shown.  I got tired of hammering my head on this wall of brick.  Further, whenever you try to search with the term coda, you get a bunch of hits written in italian.  btw, there is a kerboros use flag in the coda ebeuild, so it must integrate with it, no?

 

Dead. They went on to work on Intermezzo.

 *tecknojunky wrote:*   

>  *KermitTheFragger wrote:*   Intermezzo
> 
> http://www.inter-mezzo.org/
> 
> Looks nice, but seems dead, or is it me? Seem to be quite dead.  There's Lustre that is suppose to take its place, but it seem that the development is stalled.

 

Don't know current Intermezzo status, but I'll choose this point to give a little history of Carnegie Mellon University. AFS got its start there, and some number of CMU-ers took it with them and founded Transarc, which marketed AFS, and eventually Transarc was bought by IBM. It then withered for various reasons, and eventually the code base was released as OpenAFS. (If I Understand Correctly)

In the meantime, CMU still had comp sci professors and students who wanted to work on distributed filesystems, hence Coda. Eventually more changed, and (again, IIUC) that line of work turned into Intermezzo.

 *tecknojunky wrote:*   

>  *KermitTheFragger wrote:*   CIFS
> 
> http://linux-cifs.samba.org/
> 
> Beter known as the SMB protocol. Stable. Seems like a viable alternative to NFSv3. I'd like to be able to make samba authenticate Windows users (PDC I think it's called.  Or is it ADS?  Oh well, I'm so not Windows anymore... woohoo!  ).  Is this samba-tng that does that?  I know it's a mix of ldap and kerboros, not much more.

 

I have one (dual boot) Windows box at home, and when my son comes home for the summer there will be a second, so I have some desire to get Samba running. At the moment, I have Windows printing directly to CUPS, since it does support IPP. It took an odd assortment of stuff to get Win98SE to do this, but I understand that my son's XP will "just work." (yeah, right.) In the meantime, presuming I get my LDAP/Kerberos/SASL soup cooked, I'll add Samba to that and make a PDC. But I'd still rather have nfsv4 for Linux-Linux file service, since CIFS is even less Posix-ish than afs. (No symlinks, for one)

----------

## tecknojunky

 *depontius wrote:*   

>  *tecknojunky wrote:*    *KermitTheFragger wrote:*   NFSv3:
> 
> Solid, Simple, but insecure, wouldnt want to use it anywhere except for home use. Takes only one smart pants with a knoppix cd to mess things up. So far, it's the only one that has suited my needs.  It's relatively simple and fast.  The drawback... if it's used for /home and it goes down, everyone get frozen. 
> 
> Scares me from a security standpoint. I've seen a little deployment at work, and a decade ago I configured the OS/2 NFS client, including setting UID and GID in a file. THAT'S what scared me to begin, and my impression never got much better. Even though I'd just be running my home network, I still try to use best practices. Right now I'm in the process of bringing up the LDAP/Kerberos/SASL soup, wading through generating certificates that OpenLDAP will be happy with.
> ...

 Well, you've seen posts/bugs from me because I did struggle to get it working.  In fact, I think I jump rather to quickly on the train.  The kernel was ready for NFSv4 but the tools and the distribution were not.  For proof, it took me nearly a month to realize that to beneficiate from it, you must mount with the nfs4 type option.  :Rolling Eyes:   I study/work in a university and they were using NIS for password checking, and with Linux getting less and less marginal, smart asses who sniffs passwords started to be a problem, hence why we were looking to use NFSv4 security features.  Finally, we ended up still using NIS, but inside a IPsec pipe.  Still, NFSv4 is setup on a cluster that I'm setting up, but I don't have the time to finish the configuration (mainly learn how kerberos works).  I guess I have to wait for somebody to write a nice howto.  :Wink: 

 *depontius wrote:*   

>  *tecknojunky wrote:*    *KermitTheFragger wrote:*   AFS
> 
> http://openafs.org/
> 
> Great features (client side catching, Kerberos support, etc), Solid, but HUGE to setup and maintain. Seems only usable when you have 100 > users. I would have tried it since there's a nice Gentoo howto, but there's no support for 2.6 kernels. 
> ...

 There are also GFS ebuilds that have recently been added into portage.  I've read a little bit about it and it seems to be like acheiving the same result as Coda, ASF and Intermezzo, but thrue LVM.  I don't remember specificaly how the rights are managed, but I think it uses the unix style.  I'm very tempted to give it a shot with two nodes on the cluster.

 *depontius wrote:*   

>  *tecknojunky wrote:*    *KermitTheFragger wrote:*   CIFS
> 
> http://linux-cifs.samba.org/
> 
> Beter known as the SMB protocol. Stable. Seems like a viable alternative to NFSv3. I'd like to be able to make samba authenticate Windows users (PDC I think it's called.  Or is it ADS?  Oh well, I'm so not Windows anymore... woohoo!  ).  Is this samba-tng that does that?  I know it's a mix of ldap and kerboros, not much more. 
> ...

 I've read a bit more on that and I'm in the process of setting up a PDC that will get the users and passwords from the OpenLDAP server.  I know, I know.  No big deal there. But you gotta love networking.  So much fun, all the stuff you can do.  Endless. :Very Happy: 

----------

## depontius

 *tecknojunky wrote:*   

>  *depontius wrote:*    *tecknojunky wrote:*    *KermitTheFragger wrote:*   NFSv3:
> 
> Solid, Simple, but insecure, wouldnï¿½t want to use it anywhere except for home use. Takes only one smart pants with a knoppix cd to mess things up. So far, it's the only one that has suited my needs.  It's relatively simple and fast.  The drawback... if it's used for /home and it goes down, everyone get frozen. 
> 
> Scares me from a security standpoint. I've seen a little deployment at work, and a decade ago I configured the OS/2 NFS client, including setting UID and GID in a file. THAT'S what scared me to begin, and my impression never got much better. Even though I'd just be running my home network, I still try to use best practices. Right now I'm in the process of bringing up the LDAP/Kerberos/SASL soup, wading through generating certificates that OpenLDAP will be happy with.
> ...

 

Can't do it tonight, because we just picked my son up at the airport for Spring Break. But I have a couple of HowTo's I can point you to. I've set up MIT Kerberos a few times and have had it running successfully. At the moment I'm fighting SSL certificates so I can get my schemas loaded into LDAP. Then after that it's time to get Heimdal Kerberos up and running. Turns out Heimdal is prefererable in my situation, because it's thread safe (The most recent MIT release is better, but not all the way there.) and can store its passwords using LDAP as the database. Turns out that the latter is good when you go add Samba, because it lets the whole shebang integrate, rather than having to keep at least 2 password databases in sync.

But for the moment, family and woodworking (entertainment center for the living room) take priority. If you're impatient, look for "Jose Gomez Gonzales" and you'll find hime haunting the LDAP mailiinglists, with a pointer to his website and the HowTo on it. Good extra - he's doing his work on Gentoo, so it's all more directly applicable.

----------

## tecknojunky

Well, it was not a request as such.  :Laughing:  Thanks for wanting to help though.

I'm trying to mater LDAP little by little.  I've been using it for like close to two years, mainly at home for centralized username/password, but it's the kind of thing you set it and forget it, so it's always a challenge to install elsewhere as it never wants to work on the first shot.  I don't know how advance you are with OpenLDAP, but I have two advices to give you about it: 1- forget about ipv6; 2- for Windows users authentifications on the LDAP, check out pGina.  pGina is a very good alternative to setting up a PDC.

And, I agree with you.  If you son is there, if the family is there, if the wood is there, and especially if the entertainment system will be there, then the hell with this.  :Wink: 

----------

## depontius

 *tecknojunky wrote:*   

> Well, it was not a request as such.  Thanks for wanting to help though.
> 
> I'm trying to mater LDAP little by little.  I've been using it for like close to two years, mainly at home for centralized username/password, but it's the kind of thing you set it and forget it, so it's always a challenge to install elsewhere as it never wants to work on the first shot.  I don't know how advance you are with OpenLDAP, but I have two advices to give you about it: 1- forget about ipv6; 2- for Windows users authentifications on the LDAP, check out pGina.  pGina is a very good alternative to setting up a PDC.
> 
> And, I agree with you.  If you son is there, if the family is there, if the wood is there, and especially if the entertainment system will be there, then the hell with this. 

 

I forgot about ipv6 pretty much after finding it came up by default, and shutting it off. I know it's a good idea in the long run, and maybe after I get my network to "full capability" I'll start playing with it. But for the moment, I've got bigger fish to fry, and ipv6 doesn't seem to be flying in the US.

With LDAP I'm having trouble with certificates, right now. I had a little trouble with certificates starting the server, but once I got my CA cert in the right place with the right permissions, things came up. But I can't load the schema - it complains about confidentiality. Looking on the net, I tried the "-Z" flag on ldapadd, and then it complains about my certificates. I thought having LDAP accept the certificates when it started would be sufficient. But NoooOOOOoooo! I began by using a thing called "roCA", a combination of OpenCA plus Knoppix on a CD. Gives a bootable system, and stores data on flash. Effectively like having a dedicated computer to run your CA - when needed. Then I saw some fairly simple CA/signed-cert instructions on the OpenVPN website, and tried it that way. Now I've found an OpenSSL HowTo that includes CA and signing, and has some configuration tweaks that neither of the other two appeared to. (Maybe I missed them on roCA, since it was my first time trying that stuff.) I haven't had time to try the OpenSSL HowTo, yet. Years back, I had UWash imaps running with self-signed certs, but I'd heard that OpenLDAP really doesn't like them, and I kind of liked the idea of having my own CA, anyway. It's another learning curve.

3 of 4 door edges sanded. Then I have to go back and compensate for the fact that a few things aren't perfectly square. For most purposes it's close enough, but for doors it isn't. This is my first time on a project this big, first time I've made a wood project with doors, etc. More learning experience. Unfortunately the entertainment system isn't there - we're getting the cabinet done first, and then the 19-inch "Chuck and Di" TV goes back on top. (Chuck and Di's wedding was one of the first things my wife and I watched on it.)

----------

## RockCrusha

for the purely security minded, what about shfs?

http://shfs.sourceforge.net/

```
About shfs

Shfs is a simple and easy to use Linux kernel module which allows you to mount remote filesystems using a plain shell (ssh) connection. When using shfs, you can access all remote files just like the local ones, only the access is governed through the transport security of ssh. Shfs supports some nice features:

    * file cache for access speedup

    * perl and shell code for the remote (server) side

    * could preserve uid/gid (root connection)

    * number of remote host platforms (Linux, Solaris, Cygwin, ...)

    * Linux kernel 2.4.10+ and 2.6

    * arbitrary command used for connection (instead of ssh)

    * persistent connection (reconnect after ssh dies) 
```

----------

## depontius

For the purely security-minded, I'd agree with you. The discussion of shfs came up on another thread, in the context of backups on a college campus, and given the wild nature of a campus network, I think it's an appropriate use. But shfs is built in userspace, and piggybacked on top of ssh, neither of which is bad, for occasional and security-priority use.

In my context of a network filesystem, I'd like to move toward Single-System Image, and I don't think the performance of shfs would be good enough to be used as your $HOME directory. Heck, we find things *designed* as network filesystems too slow, at work. When there is an apparent improvement moving from a 5400rpm to a 7200rpm drive, you can bet that adding a chunk of ethernet there will show up if it's in between you and most of your work. "Real" networked filesystems work really hard on performance, as well as integrity and hopefully security. IMHO, shfs is "second-mission" to too many of its components to really perform well enough for anything but specific high-security uses.

----------

## tecknojunky

I could not state a opinion on it.  Now I can.  :Very Happy: 

----------

## NotQuiteSane

I'm getting ready to build a server, and firured I should intergrate file server duties into it, most likely NFS.   I currently have 137gb spread across 9 scsi's (not too bad for about $40 invested  :Wink:  )  I'm gonna be sticking <5 drives  into the server.   I'd like to have all my drives (/ partitions, probably do multiple swap files) appear as one unit (LVM).  is this possible with NFS?  or am I (at best) looking at 2 volume groups, one local, one networked?

what about security?  I'm currently using a script to mount /home via dm_crypt, I need to RTFM and figure out how to creat the /mapper/* entries before it does vgchage -a y during boot, then I'll add in /var.  I don't see a big need to encrypt everything (tmp is tmpfs & swap is encrypted).  any problems using dm_crypt over nfs?  or am I gonna get to have lots of fun RTFMing to find a better way to do all this?

NQS

----------

## tecknojunky

 *NotQuiteSane wrote:*   

> I'm getting ready to build a server, and firured I should intergrate file server duties into it, most likely NFS.   I currently have 137gb spread across 9 scsi's (not too bad for about $40 invested  )  I'm gonna be sticking <5 drives  into the server.   I'd like to have all my drives (/ partitions, probably do multiple swap files) appear as one unit (LVM).  is this possible with NFS?  or am I (at best) looking at 2 volume groups, one local, one networked?

 

I don't quite understand the setup you want to acheive.

NFS runs no problem on top of LVM.  I don't know if it is more efficient to run separate native swap partitions or let this be managed under LVM.  I truly have no clue and be curious to know that.

For global file system, there is now a gfs specific kernel in Portage (sys-kernel/gfs-kernel).  I'd like to try it soon.  From reading the specs at Red-Hat, I kinda puts NFS (or some other similar technology) under LVM and not the other way around.  For the little I can remember, it sounded like a very elegant solution and was very appealing to me.  I suggest you go read it too.

----------

## NotQuiteSane

 *tecknojunky wrote:*   

>  *NotQuiteSane wrote:*   I'm getting ready to build a server, and firured I should intergrate file server duties into it, most likely NFS.   I currently have 137gb spread across 9 scsi's (not too bad for about $40 invested  )  I'm gonna be sticking <5 drives  into the server.   I'd like to have all my drives (/ partitions, probably do multiple swap files) appear as one unit (LVM).  is this possible with NFS?  or am I (at best) looking at 2 volume groups, one local, one networked? 
> 
> I don't quite understand the setup you want to acheive.
> 
> NFS runs no problem on top of LVM.  I don't know if it is more efficient to run separate native swap partitions or let this be managed under LVM.  I truly have no clue and be curious to know that.

 

I'm sorry, I wasn't thinking swap under lvm.   just multiple swap files on the local systems.   so if there's 5 drives in the server, it'll have 5 swap files (say partition sd{a,b,c,d,e}1 where sd{a,b,c,d,e}2 will then be a lvm partition.

as to how well it works, I'm running 3 swap partitions on this box, and all I can say is "WFM".  my little brother who "knows more about linux than I" reccomended it, so i tried it

I quick search bought up this:

 *Linux: Using Multiple Swap Partitions In 2.4 wrote:*   

> They were talking about a large server with a lot of disk drives, and you could put a swap partition on many of them, and set all these swap partitions to the same priority. This way they would work more like they were in a RAID setup, and the speed of swap writing and reading from the disks would be improved.
> 
> Also, near the end of the replies to the 'How to use RAM as Swap' article, there was mention that someone should be reworking the swap algorithm so it didn't use such a simple and slow search method for finding and using swap slots. I believe it was a mention of Andrew Morton himself saying something like that in an lkml email. I don't know if that has been done, but the 2.6 kernel swap is a lot different than the 2.4, so maybe it was at least attempted.

 

So it looks like i need to go STFW, as with a 2.6 kernel, multiple swap partitions may be useless

as to the other side of the equation, what I am asking is can it be configured so all physical volumes, regardless of machine physically installed in in the network, all belong to the same volume group?   or will each machine have to have it's own VG?

 *Quote:*   

> For global file system, there is now a gfs specific kernel in Portage (sys-kernel/gfs-kernel).  I'd like to try it soon.  From reading the specs at Red-Hat, I kinda puts NFS (or some other similar technology) under LVM and not the other way around.  For the little I can remember, it sounded like a very elegant solution and was very appealing to me.  I suggest you go read it too.

 

I looked at it.  i need to read some more.  I would really like to use ZFS, but I'm not sure if there'll be a non-sun version (or if the sun version is even being used in any production enviroments yet).  but (speaking as a person who does this as a hobby), ZFS is how things "should be" that plus automatic processor clustering (network performs as a muti-chip computer, adding a box to the network increases performance across the entire network)

NQS

----------

## tecknojunky

 *NotQuiteSane wrote:*   

> I'm sorry, I wasn't thinking swap under lvm.

 I tought so.  :Wink: 

 *NotQuiteSane wrote:*   

> I would really like to use ZFS...

 Hooooo, nice.  Thanks for nurishing my brain.

 *NotQuiteSane wrote:*   

> ...but I'm not sure if there'll be a non-sun version (or if the sun version is even being used in any production enviroments yet).

 Well, according to this: *IBM wrote:*   

> ... zFS is designed to (...) be built from commodity, off-the-shelf components (PCs, object store devices) and a high-speed network, and run on existing operating systems such as Linux.

 

----------

## KermitTheFragger

Nice to see the thread is still alive and kicking, and there are more similar minded (Network-filesystem-philes   :Smile:  ) like myself.

Seems openAFS is making good progress:

http://www.openafs.org

https://forums.gentoo.org/viewtopic-t-242362.html

https://forums.gentoo.org/viewtopic-t-331633.html

The "unstable" release is being used by a number of people in production enviroments without major problems. Also its expected to hit the big 1.4 any time soon now. 

I haven't had time to evaluate the current state of NFSv4. How would you guys rate it's current feature set (but also stability) against openAFS ?

Also CODA seems to make some progress:

http://www.coda.cs.cmu.edu/news.html

----------

## tecknojunky

 *KermitTheFragger wrote:*   

> Seems openAFS is making good progress:
> 
> http://www.openafs.org
> 
> https://forums.gentoo.org/viewtopic-t-242362.html
> ...

 Does it work on the 2.6 kernel now?

 *KermitTheFragger wrote:*   

> I haven't had time to evaluate the current state of NFSv4. How would you guys rate it's current feature set (but also stability) against openAFS ?

 I'm still using nfsv4 in a v3 fashion  :Sad: 

 *KermitTheFragger wrote:*   

> Also CODA seems to make some progress:
> 
> http://www.coda.cs.cmu.edu/news.html

 Another one i've tried but never had success configuring properly.

----------

## KermitTheFragger

 *tecknojunky wrote:*   

> Does it work on the 2.6 kernel now?

 

Sure does! I'm using it, and it seems stable.

 *tecknojunky wrote:*   

> Another one i've tried but never had success configuring properly.

 

Me neither. Also development seems to be going a lot slower with Coda then with openAFS.

----------

## chris.c.hogan

I've been playing with NFSv4 for a day or two and have some questions. As this thread is about network file systems, it seems a good place to post.

First my setup. The system was originally setup for NFSv3. I have several partitions set up, among them are /srv and /home. The entire /home file system is exported. The /srv has several subdirectories. Of them, only /srv/nfs is exported. I'm currently using TLS encrypted LDAP for authentication. I've thought about using Kerberos V, but one experiment at a time... For simple security, I'm using subnet exports.

So, playing around with NFSv4. One of nice things about NFSv4 is having a root defined (using fsid=0 on exported NFS root). One of the things I hated about NFSv3 was needing the full server path on the client (mount server:/srv/nfs/public /mnt/public vs. mount server:/public /mnt/public). This works well for the /srv/nfs root export, but not for /home. So I unmounted /home and remounted it as /srv/nfs/home. I also created a symbolic link from /srv/nfs/home to /home on the server so I'd still have home directories. However, NFS won't export mounts under the NFSv4 root directory without the nohide option. The exports man page has this to say about nohide:

 *Quote:*   

> The nohide option is currently only effective on single host exports. It does not work reliably with netgroup, subnet, or wildcard exports.

 

Is this still true?

One other question: are statd and lockd required on the client side in a NFSv4 setup, or is it just for NFSv3? I get the impression that locking works a lot different in v4.

From http://www.ietf.org/rfc/rfc3530.txt

 *Quote:*   

> 1.4.5.  File locking
> 
>    With the NFS version 4 protocol, the support for byte range file
> 
>    locking is part of the NFS protocol.  The file locking support is
> ...

 

Thanks for any insights!

----------

## Redeeman

nfs3 can be fairly secure on an intranet, (or over internet via vpn).. you just gotta enable squashing..

----------

## roguetoad

any word on nfs4 development? developer pages seem pretty devoid of any action.

anybody have it actively deployed on their network?

----------

## drescherjm

I am very interested in this discussion so I am waking up this old thread... I have several different needs for a distributed filesystem which are: 

1) I want to be able to add storage to the filesystem by adding additional servers (Athlon64 6 to 10 drive RAID6 servers with > 1TB of raid disk space) and in this case I want this additinal storage to be as transparent to the user as possible. I do not want them to have to manage what server to move their data to and a single folder should be able to span servers. 

2) I want to be able for a server to go down and the filesystem still stays up only that any files on the server that went down will not be available but the network should not be experiencing constant timeouts when 1 or more servers are down.

3) It is a must that no data is lost (as there is a long delay between generation and backup) so the system must be very reliable.

Currently we use nfs on our linux servers and have both windows (samba) and linux (nfs) clients with most of the users being on the windows side. This represents several problems as when we add storage we insert a mount point inside the tree and which samba is able to see however nfs needs each client to explicity mount the share and this says nothing of the juggling of storage data needed to be done to ensure none of the servers are full of data and since the windows clients get back the free space on the root filesystem they have no idea how full or where any of these mount points are.

I am looking right now at OpenAFS, Coda and gfs. At first look OpenAFS and Coda are out and I am not sure about gfs. I rule OpenAFS out because it may be too compilcated and it fails #1 and possibly #2. I am not sure Coda is a good fit because it has if I read correctly extreamly small limits on the number of files in a folder and in the whole disk. Right now we have 5TB of online data (3 servers and 4 mounts) in like 10^6 files some folders will have 20,000 files of 1 MB each. Also the docs for Coda are way out of date. I am not sure at all Coda will handle 1TB of data as all the docs mention sizes in a small number of GB. And then the docs even question its stability...

----------

