# RAID over NFS

## slick

This is a HowTo to create a RAID over NFS

(this post in german)

 :Exclamation:  It's not for common use, it's very experimental! 

We have a server who mount the RAID and (in this case 3) clients who be part of the RAID.

Install nfs-utils on server and clients.

```
emerge nfs-utils 
```

create a shared directory and start the nfs-server on every client (example for trusted network)

```
mkdir /share 

echo "/share 192.168.0.0/24(rw,async,no_root_squash)" >> /etc/exports 

/etc/init.d/nfs start
```

On the server install raidtools (and prepare the kernel to use RAIDs)

```
emerge raidtools
```

now create mountpoints on the server and mount the shares from the clients

```
mkdir /mnt/client1 

mkdir /mnt/client2 

mkdir /mnt/client3

/etc/init.d/nfs start 

mount host1:/share /mnt/client1 

mount host2:/share /mnt/client2 

mount host3:/share /mnt/client3 
```

now create "container-files" on the clients (in this case for RAID5, all with the same size)

```
dd if=/dev/zero of=/mnt/client1/container bs=1M count=500 

dd if=/dev/zero of=/mnt/client2/container bs=1M count=500 

dd if=/dev/zero of=/mnt/client3/container bs=1M count=500 
```

now create a loop-device for every container-file

```
losetup /dev/loop1 /mnt/client1/container 

losetup /dev/loop2 /mnt/client2/container 

losetup /dev/loop3 /mnt/client3/container 
```

create a /etc/raidtab for the RAID (example for RAID5)

```
raiddev /dev/md0 

    raid-level                  5 

    nr-raid-disks               3 

    nr-spare-disks              0 

    persistent-superblock       1 

    parity-algorithm            left-symmetric 

    chunk-size                  128 

    device                      /dev/loop1 

    raid-disk                   0 

    device                      /dev/loop2 

    raid-disk                   1 

    device                      /dev/loop3 

    raid-disk                   2 
```

create the RAID (and watch until is finish)

```
mkraid /dev/md0 

watch cat /proc/mdstat 
```

create a filesystem and mount it

```
mke2fs /dev/md0 

mount /dev/md0 /mountpoint 
```

Finish

------

small Performance-Test (RAID5, 3 Clients, 100MB LAN)

```
# tiotest

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write          40 MBs |    6.3 s |   6.308 MB/s |   0.2 %  |  13.7 % |

| Random Write   16 MBs |    6.0 s |   2.602 MB/s |   0.1 %  |   4.4 % |

| Read           40 MBs |    0.1 s | 268.125 MB/s |   6.7 %  |  92.5 % |

| Random Read    16 MBs |    0.1 s | 244.500 MB/s |   6.3 %  |  89.2 % |

`----------------------------------------------------------------------'

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        0.127 ms |      327.244 ms |  0.00000 |   0.00000 |

| Random Write |        0.107 ms |       66.123 ms |  0.00000 |   0.00000 |

| Read         |        0.014 ms |        0.795 ms |  0.00000 |   0.00000 |

| Random Read  |        0.014 ms |        1.582 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.068 ms |      327.244 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'
```

Is one client goes down all prozesses who read or write on the RAID will "freeze" ...

----------

## rojaro

Crazy Idea, but this might be kind of usefull when used with RAID1 .. this would provide a pretty fault tollerant online Backup ... if it just wouldnt be so slow ...

----------

## slick

sure, RAID5 was only an example, but the problem is that the RAID does not known if a client offline. He "only" knows that /dev/loopX ist present (or not), not that the file behind /dev/loopX is not available. In this moment the RAID does only "freeze" and do not work like with normal disks.

----------

## Forse

I agree about crazy idea, but since gigabit nics and switches/hubs are cheap nowdays...I find this to be useful idea.

----------

## -Craig-

Couldn't you ping all the computers periodically and remove /dev/loopX if the PC serving that is not available anymore?!

BTW: It's a crazy idea  :Wink: 

----------

## slick

I think you must remove the /dev/loopX before the PC goes offline, otherwise it is inpossible to remove it because the raid crash. But I didnt try this in every case. Try it and report us.

If you looking for an filesystem like raid0 over LAN try MapFS, I read his can do like this. But I dont know how stable it is or details about it.

 *Quote:*   

> MapFS implements a Linux filesystem which utilizes copy-on-write functionality and existing Linux filesystems to allow component filesystems (or portions thereof) to be combined into a single virtual filesystem that appears to be fully writable. 

 

----------

## NewBlackDak

Wouldn't this work better with ATA-over-ethernet?

If you could get rid of the loopbacks, and have a couple setup as online spares then loosing a network node "in theory" should let you work along without missing anything.

----------

## den_RDC

Well, EVMS has cluster/network features etc, so i think this could be setup way easier with EVMS, probably with a slightly better redundancy ( if the raid array freezes if 1 of the disk fails, you actually increase your risk of losing access, so the R in raid isn't really applicable anymore )

----------

## Mad Merlin

Cool idea, I'm just playing with this now, except using sshfs rather than nfs, works quite well.

----------

## adenum

 :Very Happy: 

Hi every body

this night I ve been thinking of RAID over NFS

I think we should take a raid 0 over NFS

the perfs might be as good as a miror during rebuild

-no test for the moment-

imagine a free service

I gave you back 1 Go totally secured space for free

you gave me by sharing 20 Go to make the service free

otherwise you rent it for 1 $ / year

consider the level of security for grappes made ith 10 disks

the first is your local one

the 9 others are balanced with the 8 bits / byte and a parity bit then 9 + 1 disks where your datas are balanced

in level 2 we need 21 disks

in level 12, 22 527 shares it is possible

your data is secured

1	0

10	1

21	2

43	3

87	4

175	5

351	6

703	7

1 407	8

2 815	9

5 631	10

11 263	11

22 527	12

45 055	13

90 111	14

180 223	15

360 447	16

720 895	17

1 441 791	18

2 883 583	19

5 767 167	20

11 534 335	21

23 068 671	22

46 137 343	23

92 274 687	24

184 549 375	25

369 098 751	26

738 197 503	27

1 476 395 007	28

2 952 790 015	29

5 905 580 031	30

imagine a bussiness plan if we offer up to level 2 for free

and make pay the others level

we could be rich ...

Who wants to participate to that crazy idea ?

let me know

----------

## stobbsm

That may be just crazy enough to work out....

I'll run some VM tests and see what comes back...at least as far as stability is concerned.

----------

## John R. Graham

Tahoe Filesystem is probably a much better way to go.

- John

----------

## thejbo

GlusterFS is probably a better solution. 

It's a great way to get a distributed, fault-tolerant file system without the expense of a SAN.

The latest version has recently been added to Portage:

http://gentoo-portage.com/sys-cluster/glusterfs

----------

## richard.scott

 *NewBlackDak wrote:*   

> Wouldn't this work better with ATA-over-ethernet?

 

Yes, ATAoE works much better... it even notices when a node goes down and doesn't just hang like this solution  :Smile: 

----------

