# iSCSI vs. NBD vs. AoE

## wildhorse

Has anybody compared iSCSI vs. NBD vs. AoE or knows about such comparison done elsewhere?

I am most interested in effective bandwidth (transfer rate), latency and reliability.

I am planning to use only a LAN (1Gbit/s ethernet).

iSCSI - Internet SCSI

NBD - Network Block Device

AoE - ATA over Ethernet

----------

## alex.blackbit

i have no practical experience with all 3, but a few thoughts about them.

scsi is feature fuller then ata in principal. and i do not believe that you would benefit from having a raw block device on the remote machine.

so i guess iscsi would be the best choice.

just my .02$

alex

----------

## eno2001

I recently was looking for the same answers you are because I'm building a poor man's "SAN" here at home.  What I found in my Google searches that helped me choose iSCSI (although there are reasons I don't like iSCSI) was a forum post written by someone who owns a storage technology company.  His opinion was that if you care about your data (which I do to an extent), you don't want NBD because while it is free of the overhead of iSCSI (TCP/IP adds a performance hit) it has some severe limitations.  The largest one being that it can't handle the loss of network connectivity gracefully. This could result in data loss or corruption.

AoE had a little more going for it in terms of surviving a network outage, but it lacks the error correction of TCP/IP that iSCSI has, it's not a recommended solution if you care about your data.  I'll not that I didn't really have any interest in AoE, so I didn't delve further to determine if this criticism is 100% valid.  He also noted that he would not use anyone's implementation of iSCSI that didn't make use of crc32 cryptography.  I think this is likely related to network security.  That topic is debatable since for someone like me who is only dealing with virtual machine images in a home setting, and I have a dedicated storage network that is isolated to three machines, security is not a concern.  At work... it certainly would be a big deal, and the person who wrote that comment seemed to imply that AoE and NBD are completely lacking in that area.  

Since I plan to use this "SAN as a way of hosting virtual machine images for the Xen hypervisor, I chose to go the same route that VirtualIron did and make use of iSCSI.  I played around with NBD for a while, and I'd love to see it extended beyond what it is.  Namely the IOCTLs...  I'd love to be able to share a CD or DVD writer over the network.  Apparently this is not possible with NBD.  I have successfully shared a DVD drive for DVD playback over WiFi though...  :Smile: 

 *wildhorse wrote:*   

> Has anybody compared iSCSI vs. NBD vs. AoE or knows about such comparison done elsewhere?
> 
> I am most interested in effective bandwidth (transfer rate), latency and reliability.
> 
> I am planning to use only a LAN (1Gbit/s ethernet).
> ...

 

----------

## wildhorse

Eno2001, thank you for your nice comments!

Reliability is an issue for me because I am dealing with tens of terabytes (TB).

So your point about iSCSI vs. AoE or NBD is well taken. Latency is another important issue for me. Since NBD is based on TCP/IP and lacks reliability, it seems that NBD is no option for me.

I think AoE is still an option because it gives me a minimum of latency. Every microsecond of latency will limit the bandwidth of my complete solution. But I appreciate your comment and will take a close look at the checksum situation. I cannot rely on the small 32-bit ethernet checksum. But something like md5 might be overkill. Perhaps I add my own checksum code (in Assembler) to the AoE routines.

BTW, VMS http://www.hp.com/go/openvms/ is able to access any device in a cluster like a local device. I do burn disks across the lcoal network. I have even transferred data from a tape drive across an IP network (via DECnet-Plus over IP) between two buildings. And all that is a available since more than two decades. Too bad that there is no MSCP available for Linux.   :Wink: 

----------

## alex.blackbit

oh, i see you are familiar with some computing techniques that seem to get forgotten these days.

(open)vms is indeed a very reliable platform for doing data processing.

with one crying eye i remember the days when "clustering" meant "software fault tolerance" like in SFT III of novell netware or when a "next generation" filesystem was "FILES-11".

this post will not help you, it is my expression of sentimentality.

----------

