# Cluster & distributed filesystem options

## expat_iain

Am looking into the potential for installing a cluster with a distributed filesystem that is transparent to the end user. The idea is to have a few machines sharing load and also providing redundancy so that should a machine fail, the others will continue to serve without interruption to users.

The options currently seem to suggest something like openMosix for the clustering and AFS for the filesystem distribution. There are a couple of points that I'm not too sure of

- There seems to be a size limit on AFS partitions

- AFS partitions require ext2 formatting

- 2.6 users have reported problems with AFSDoes anyone have experience or recommendations towards implementing such a solution?? It seemed to be a better idea to check here first before trying to reinvent the wheel.

Regs.

Iain.

----------

## setagllib

NetBSD 2, UFS w/ softdep, AFS on top of that

Linux isn't always the answer. Definitely not if you were planning on making this cluster as reliable as possible.

http://www.madscientistresearch.net/ProjectList/ProjectNorton/AfsOnNetBSD (an article that may or may not help you, but is better than nothing)

----------

## expat_iain

Okay, I see that BSD can take the AFS side. Would you say that this is the correct solution for what is trying to be achieved??

Have you implemented such a solution with the clustering aspect included?

Regs.

Iain.

----------

## setagllib

It doesn't sound like you *need* clustering if it's just a distributed file server. It sounds like AFS can do that on its own.

No, I haven't done anything like this, I just know NetBSD makes a very lovable server, and happens to work with AFS.

----------

## expat_iain

Been doing some hunting and am starting to think that something that mixes clustering and GFS may be the way to go. GFS seems to support the disributed filesystem in that I can lose a server and the filesystem integrity is still there. If I can then get a cluster to appear as one machine then I can finally get rid of the last M$ boxes here.

Anyone been trying this mix??

Iain.

----------

## feffi

Hey Guys,

I'm just working on this type of project for my studies. I'm trying to build up a server cloud (no processing cloud) of fully transparent servers with a distributed and redundant filesystem.

Maybe I can post some news in Jan. 2007.

Currently I'm working exactly on this problem an found some good answers:

http://sourceware.org/cluster/gnbd/

https://open.datacore.ch/DCwiki.open/Wiki.jsp?page=GFS.GNBD.Usage

https://gentoo.datacore.ch/DCwiki.gentoo/Wiki.jsp?page=Main

If you got any good ideas, just pm me or post here...

----------

## GNUtoo

i don't know how gfs work but it has been integrated in the 2.6.19 kernel

----------

## feffi

 *GNUtoo wrote:*   

> i don't know how gfs work but it has been integrated in the 2.6.19 kernel

 

GFS has a severe design flaw. You can't mount -o loop. This disabled SSI effectively.

----------

## feffi

Hoorray or Horrible?

Today I came across a very interesting site: GlusterFS

It has very features and integrated load-balancing:

 Designed for O(1) scalability and feature rich.

 Aggregates on top of existing filesystems. User can recover the files and folders even without GlusterFS.

 GlusterFS has no single point of failure. Completely distributed. No centralized meta-data server like Lustre.

 Extensible scheduling interface with modules loaded based on user's storage I/O access pattern.

 Modular and extensible through powerful translator mechanism.

 Supports Infiniband RDMA and TCP/IP.

 Entirely implemented in user-space. Easy to port, debug and maintain.

 Scales on demand. 

Scheduler:

 alu : ALU means Adaptive least usage, uses an algorithm to distribute load.

 nufa : Non-Uniform Filesystem Scheduler.

 random : Randomly schedules the file to one of the child node.

 rr : Round Robin Scheduling. 

Interconnects:

 ib-verbs: Uses Infiniband verbs layer for RDMA communication. Fastest interface.

 ib-sdp: Uses Infiniband SDP (sockets direct protocol) for RDMA communication.

 tcp: Uses regular TCP/IP or IPoIB interconnect. 

I will implement and test this with ocfs2 and GlusterFS in the eden of march   :Twisted Evil: 

Here's a tutorial: GlusterFS Configuration Example for Four Bricks

And also there is a distribution for it (but I've not read far from there): Gluster - The GNU Cluster Distribution

Have fun and if you try it, give me a hint   :Smile: 

----------

