# AOE :: ATA-over-Ethernet on Gentoo

## MrUlterior

Warning, this post is a collection of thoughts & questions on AOE / ATA-over-Ethernet on Gentoo. There's no specific problem I'm encountering, even though the packages are masked everything seems to work so far.

The story so far ...

I'm sick of buying hard drives and stuffing them in workstations everywhere to give me redundancy & RAID, I have a dedicated file server & it seems to me that should be it's purpose -- I want all my workstations to be diskless and mount RAID volumes over the network from the fileserver. I want to recuperate all the disks in all my workstations for addition to my LVM/RAID on my file server.

So, last night drunken & euphoric after a few drinks & reading about AOE -- I downloaded aoe, aoetools & vblade & installed them on 3 gentoo boxes:

A x86 Server (Athlon XP, SATA 1TB storage in MDADM software RAID-1)

B x86 Client (Intel PIV Xeon, Client)

C amd64 Client (Athlon 3000 64bit, Client)

Obviously vblade was only installed on (A) - both (A) & (B) were not able to modprobe aoe,

to fix this, I had to update 2.6.12-gentoo-r10 to a vanilla 2.6.13.2 kernel & rebuild aoe, however - strangely enough the same 2.6.12-gentoo-r10 kernel worked fine on the amd64 (C) box

Questions

Now I know I could export several blades from loopback files on server (A) and create a RAID array on the client with MDADM combining several of these -- however these seems inefficient as the data is going to be "written" to the each disk over the network - so using double the bandwidth and require the processing power of the client. This is how all the examples I've seen do it.

Doesn't it make more sense to create the loopback or crypted volume on a RAID partition on the server (C) directly and just mount that as a filesystem directly on the clients?

I intend to benchmark this in the following days, but I just thought I'd sound out other's opinions.

Anyone else have any thoughts on AOE? It's my intention that every client will have dual teamed/bonded gigabit connectivity to the file server to optimise on performance.

For those interested, here're the references I used:

http://www.coraid.com/support/linux/

http://www.linuxjournal.com/article/8149

http://en.wikipedia.org/wiki/ATA-over-Ethernet

----------

## MrUlterior

FYI ... I've also found a Gentoo HOWTO here:

http://hilli.dk/howtos/ata-over-ethernet-aoe-in-gentoo-linux/

----------

## MrUlterior

Come on .. there *has* to be someone out there using AoE -- any insights feedback would *REALLY* be appreciated ...

----------

## ian!

Maybe this thread gets more attention in the N&S forum.

Moved from Gentoo Chat to Networking & Security.

----------

## vputz

> there *has* to be someone out there using AoE

Heh--actually, I'm waiting to see how useful it will be before jumping in, but I've been rolling the idea around for a while.  I just haven't found a compelling reason to do so (I have one box, my server, with several disks in it, but no astonishing amount of storage; a "five-disk SCSI raid-5 array" sounds cool, but when each disk is only 5G, it's just a really big 20G disk, which is a little sad in this day and age.  It does have a 2-disk raid-0 IDE array, which still only amounts to about 190G or so, so I could replace all seven disks with only a couple hundred bucks).

And I guess that's the problem, and why I haven't messed with AOE yet.  I too have a single fileserver, but on a mixed win/linux network, so I export some volumes via samba.  But it doesn't mean a vast improvement to break my fileserver into a samba server serving up an AOE disk.

> these seems inefficient as the data is going to be "written" to the each disk 

> over the network - so using double the bandwidth and require the processing 

> power of the client. This is how all the examples I've seen do it. 

> Doesn't it make more sense to create the loopback or crypted volume 

> on a RAID partition on the server (C) directly and just mount that as a 

> filesystem directly on the clients? 

I think you meant "the server (A)" and export that as an AOE device.  And yes, this makes sense, and to me is a more correct way to do it--the client would just see a single block device, without any knowledge of it being raid or wotever.  That means the clients get to be simple ("oo!  A disk, everyone!") and do simple AOE transactions, which are raided on the server side.

Good so far.  So I think your basic idea of having one beastie be the Big Giant Storage Server which runs all its disks LVM'd with wee partitions is good, and then you export the partitions/LVMs as various shelf.slot etherd devices.  Adding a new computer?  Great; create a new LVM, export it, wire it up.

Diskless workstations, eh?  OK.  How are you going to boot them, PXE or some such?

Also, it sounds like you can't have multiple computers share a block device, so there's that frustration.  This may or may not concern you, although there may be some way to have each workstation own its own exported AOE partition and then have your file server serve out (via NFS or Samba) a shared storage partition, if that's desired.

Or, if you want REAL fun, you could try installing a clustering filesystem (openGFS or some such) on a big exported partition and have THAT accessible via AOE and bypass the regular (NFS/samba) sharing altogether.  That'd be cool.  But I have no idea on workability or performance.

If my Freevo box was wired instead of having to talk via 802.11g, I might have tried it already to consolidate storage.  But sadly, it's not to be.  I'm curious to see how your project works, though--it sounds pretty doable.  I'm about to assemble the Lamest Cluster In The World (about 5 rackmount systems with scorching Pentium III/500 chips in it, so with all of them up and running I might get half the performance of my desktop system), and I may try AOE/OpenGFS on that just for my own education.

-->VPutz

----------

## vputz

...poking about on the (sparse) aoe-tools mailing list, one fellow surmised that vblade was more efficient when striped over AOE, so that you could possibly get more throughput with multiple virtual aoe devices.  I'm not sure I agree, though--Coraid sells both types of hardware (one AOE connection per disk OR one connection for a box that RAIDs together multiple machines).

Probably no way to tell but to try.

-->VPutz

----------

## MrUlterior

Thanks for your reply, I was beginning to think no one was using/interested in AoE out there  :Smile: 

 *vputz wrote:*   

> >
> 
> Diskless workstations, eh?  OK.  How are you going to boot them, PXE or some such?
> 
> 

 

Actually I was more thinking about a simple boot CD containing the /boot with the kernel isolinux & bootloader config + the kernel modules in an initrd. The reason for this being the cheapo gigabit ethernet NICs in the clients plus I want to retain the option of sticking a HDD in quickly with some win32 variant for the (rare) occasion when I host a gaming session or similar .. At the moment I'm investigating bootable USB key (usb reader + 128mb SD card) as these seem a little better suited (plus the advantage of having a neat little switch for read only / read write which is convenient..) to the task. Essentially I want to use the Knoppixesque approach of using unionfs to bond a read only file system (CD or USB key) with a read-write one (AoE) for the client/workstations boot drive.

 *vputz wrote:*   

> >
> 
> Also, it sounds like you can't have multiple computers share a block device 
> 
> .. SNIP .. 
> ...

 

Yep, as you surmise, I don't really care that sharing is non feasible without the use of a clustered FS. The stuff that really needs sharing between computers (home dirs, music, etc) is done via Samba though I'm looking for a more perfomant replacement for that. It's been a while since I've used NFS so I may give that a go & see if it's less of an irritation than the last time.

However one element that'd be interesting is have the workstations share X.org & KDE binaries via an AoE partition for the compile once, run everywhere experience. That's a project for another day though.

 *Quote:*   

> 
> 
> I'm about to assemble the Lamest Cluster In The World (about 5 rackmount systems with scorching Pentium III/500 chips in it, so with all of them up and running I might get half the performance of my desktop system), and I may try AOE/OpenGFS on that just for my own education.
> 
> 

 

Lol, well keep us updated, it sounds like quite a project  :Smile:  Can we look forward to a post detailing it?

----------

## vputz

> Actually I was more thinking about a simple boot CD containing the /boot with the kernel 

> isolinux & bootloader config + the kernel modules in an initrd.

Sure, that would work.  I like your USB-key idea better, though (less hardware involved).  Or--here's an option if you have a bunch of CF cards lying around--CF to IDE adapters; NewEgg has some for about $10-$13, and lots of folks have old CF cards they're throwing out.  None too big, but probably enough for a kernel+initrd; enough for the AOE mount.  They don't have a read-only switch that I see, though, so your USB idea may be better; depends on whether your BIOS will happily boot from USB and what surplus hardware you have, I guess.

> However one element that'd be interesting is have the workstations share X.org & 

> KDE binaries via an AoE partition for the compile once, run everywhere experience. 

> That's a project for another day though. 

As you can imagine, that's my interest for my WeenieCluster experiment.  But it won't work well in my case--I don't have homogenous machines for the cluster (some dual p500's, some K7s, and I think an old "celeron-300-overclocked-to-450" board).  Well, it would actually work OK, now that I think of it, because I'm probably going to use a premade clustering distribution like Quantian/BCCD/ClusterKnoppix, which will have basic binaries anyway (probably won't use Gentoo for the cluster, at least not the first draft).

> Lol, well keep us updated, it sounds like quite a project  Can we look forward to a post detailing it?

Well, maybe; depends on how far I get!  I had a generous "donation" of some rackmount cases and old motherboards a friend was getting rid of (about 5-6 cases) and a telco rack, but I found a place in town that buys old discarded surplus equipment from labs, and may have a proper cabinet I can use, and another friend donated an old Cisco switch.  So I think I have almost all the hardware I need, and the switch is probably partitionable so that I could have each AOE partition on its own subnet if I needed.

And boy, with the scorching performance of those p500's, I could play fun games like a POVRay rendering race between that and my Barton 2500 desktop (lol).

But the question I can't seem to answer is "why AOE instead of NFS/Samba"?  Well, except that it's unbearably cool, of course.  But if all I would be using it for is basically "mount a partition", I could accomplish the same thing with NFS or Samba; I don't really need a raw block device for that, and it's not like I need to encrypt the partitions or anything that would be more suited to block access.  A clustered filesystem would actually be useful, but I don't know if it's worthwhile, since OpenMosix migrates things around anyway.  So I'm actually pretty puzzled about that--why am I doing this again?  Have you found any performance measures regarding AOE vs Samba?  The AOE folks like quoting 60MB/sec, but that's with good equipment and I think dual bonded gigabit ethernet connections.  Heck, at home (and I still haven't figured this out) I get faster performance rsyncing a file from my Gentoo server to my Gentoo freevo over 802.11g than I do from the same server to an XP/Cygwin machine over a wired connection, so obviously there is much here that I'm completely missing...

-->VPutz

----------

## vputz

Oh, and I think something on the coraid site shed some light on your earlier question:

 *Quote:*   

> 
> 
> Now I know I could export several blades from loopback files on server (A) and create a RAID array on the client with MDADM combining several of these -- however these seems inefficient as the data is going to be "written" to the each disk over the network - so using double the bandwidth and require the processing power of the client. This is how all the examples I've seen do it.
> 
> 

 

Check out the coraid products site (http://www.coraid.com/products.htm); they have two basic types of enclosures: the "RAID" type, with dual gigabit ethernet connections which does the raid "in-box", and the "pata etherdrive storage" set which has one board/jack per card... and they're 10/100 jacks.  So the idea on that one is to have a bunch of 10/100 connections, one per blade, to a big switch with a gigabit downlink to the client; as they put it, "aggregate speed is up to 60 MB/sec").

So unless you want to have a bunch of ethernet cards in your server box, I'd drop a gigabit ethernet card in there and raid-'em-up in-box.  Of course, I only have a 10/100 switch, more's the pity, so I think if I actually try this, I'll invest in at least a wee 8-port or some such.

-->VPutz

----------

