# FusionIO ioDrive

## Treo

Hello,

I will take delivery of a FusionIO ioDrive (80GB, www.fusionio.com) today and am hoping to get it working with Gentoo. I am wondering if anyone has tried this before, and with what results. I know that the company developed drivers for RedHat and it's derivatives, so I am wondering if those can be used in the Gentoo kernel. Again, if anyone has done this, I'd love to hear your experiences.

Thanks,

Treo

----------

## gerard27

Hi Treo,

I have no answer to your question.

But I read what this is and I am impressed.

However I read on another site the price would be around $30 per GB.

That's a "little" out of my league.

Anyway thanks for the interesting reading.

Gerard.

----------

## el-chaote

Wow... I've never heard of such a device as well, but hey... it sounds really interesting  :Very Happy:  Did you read this article? 

 *Quote:*   

> The Linux driver for this piece of hardware is pretty dodgy. Sub-alpha quality actually.

 

Doesn't sound promising, does it?  :Confused:  But at least....  :Wink: 

 *Quote:*   

> they seem to be working on it.

 

And as you said...

 *Quote:*   

> In fact there's not even anything available for Debian or other respectable Linux distros, only Red Hat and its clones. 

 

Unfortunately, I couldn't find the drivers to have a look at  :Confused: 

 *Quote:*   

>  It has its own driver for Linux completely outside of the
> 
> normal scsi/sata/sas/fc block device stack, but from the user's
> 
> perspective it behaves like a block device.

 

I guess the driver is "just" a module for your kernel, which shouldn't be to hard to install  :Very Happy:  But I'm not sure if its that eay  :Wink: 

----------

## Treo

Gerard,

well, the price is right, we'll be paying $2400 for that 80GB, but, it will be considerably cheaper than buying that quantity of RAM. The company I work for makes an appliance based on Dell PowerEdge Servers, and we heavily leverage RAM for database operations (memory tables). Right now we deploy with 16GB, but found that this is insufficient, and we really need at least 32GB, or favourably 64GB. The cost for the two are $2100 and $9600 respectively. Therefore, the $30/GB of ioDrive is more economical, with the added benefit of the ioDrive being non-volatile. I just hope I can make it work...  :Smile: 

Treo

----------

## Treo

Hello again,

so, it looks like I got this partially working, however, upon loading the driver, I get the following error:

ioDrive: iodrive_pci_probe: Found PCIe iodrive at 0000:0c:00.0...

ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 16 (level, low) -> IRQ 16

PCI: Setting latency timer of device 0000:0c:00.0 to 64

Bad IO access at port 8ffc (outl(val,port))

WARNING: at lib/iomap.c:44 bad_io_access()

Call Trace:

 [<ffffffff880316c0>] :fio_driver:ifio_35bf1.2b4223e03a1f5b03477d92a5d23b0e01878.1.1.1+0x1c/0x4c

 [<ffffffff88031803>] :fio_driver:ifio_3520c.2dd7f7b19b6d693ef10525b4f1c659ffb29.1.1.1+0x3a/0x93

 [<ffffffff8803140e>] :fio_driver:ifio_ba37b.14ee6bf7bbd744cd172d215d59f695e8347.1.1.1+0x85/0xa3

 [<ffffffff8801851d>] :fio_driver:ifio_ef6f7.11a55cc31546fce86465244a574500498d3.1.1.1+0x60/0x8f8

 [<ffffffff8027404c>] kmem_cache_zalloc+0x5c/0x80

 [<ffffffff8022c228>] __request_region+0xa8/0xd0

 [<ffffffff880172b7>] :fio_driver:ifio_29183.1114deef6a8ee75320b4aa1fb5af915f8bc.1.1.1+0x93/0x1c8

 [<ffffffff8036e238>] pci_request_region+0x88/0x1d0

 [<ffffffff880176ab>] :fio_driver:ifio_81b67.10173d07bea246518aa90d2a2bbe1073262.1.1.1+0x132/0x180

 [<ffffffff80371436>] pci_device_probe+0x76/0xa0

 [<ffffffff803e5cc0>] __driver_attach+0x0/0xb0

 [<ffffffff803e5ab7>] driver_probe_device+0xa7/0x1d0

 [<ffffffff803e5cc0>] __driver_attach+0x0/0xb0

 [<ffffffff803e5d65>] __driver_attach+0xa5/0xb0

 [<ffffffff803e4ccd>] bus_for_each_dev+0x4d/0x80

 [<ffffffff803e511f>] bus_add_driver+0xaf/0x1f0

 [<ffffffff80371660>] __pci_register_driver+0x70/0xc0

 [<ffffffff8805c025>] :fio_driver:init_fio_driver+0x25/0x8d

 [<ffffffff80244e8b>] sys_init_module+0x19b/0x18e0

 [<ffffffff80209ace>] system_call+0x7e/0x83

If anyone is versed in kernel modules, I'd love to get some feedback. The drive isn't accessible, there are more errors just like this, I can provide those if there is interest. What I can tell is that it has something to do with the iomap.c that comes with the gentoo kernel, but I am not exactly a kernel programmer to really understand more of this.

Any advise would be helpful.

Thanks,

Treo

----------

## el-chaote

Unfortunately, I don't understand anything from kernel programming  :Confused:   :Very Happy: 

However, what does dmesg say after having loaded the module?

----------

## Treo

This is what it looks like after all the errors above, basically, it creates the device, but it's not accessible.

```

ioDrive: Firmware version 4294967295 (0xffffffff)

ioDrive: 0000:0c:00.0 Resetting NAND controller

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 0 failed to reset NAND

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 1 failed to reset NAND

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 2 failed to reset NAND

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 3 failed to reset NAND

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 4 failed to reset NAND

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 5 failed to reset NAND

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 6 failed to reset NAND

ioDrive: 0000:0c:00.0 Hung operation tag: 1 completed: -1

ioDrive: 0000:0c:00.0 Physical bank 7 failed to reset NAND

ioDrive: 0000:0c:00.0: Failed to successfully initialize any banks

Trying to free already-free IRQ 0

ACPI: PCI interrupt for device 0000:0c:00.0 disabled

iodrive: probe of 0000:0c:00.0 failed with error -1

ioDrive: init_fio_blk: Fusion Block Device loading...

ioDrive: fusion: registered: fio-blk

```

----------

## Treo

Just to give everyone an update.

I never did get this working in Gentoo, I got pretty darn close, but no dice on actually making it work, keeps segfaulting during modprobe. I did get it to work in Ubuntu and we were able to test its capeabilities.

Long story short, the drive is amazing, it operates at about 60-70% the speed of RAM, which is what we were looking for to store real MyISAM tables on it, not just MEMORY tables with terrible indexes.

There will soon be a Gentoo driver as well, the company I work for has comitted to a large enough order to warrant a custom driver development by FusionIO. So if anyone ever wants to try this drive on Gentoo, or will need to make it work one day, there will be driver, thanks to me in part  :Smile: 

I will post again once we actually have the driver and maybe post some more test results.

Treo

----------

## hvengel

Treo.

Any word yet on updated drivers?

----------

## Treo

Hi,

actually, yes, we've gotten a customized driver that works with some Gentoo kernels, specifically, 2.6.22-gentoo-r5. That's the production kernel we use right now, so that's what their driver has been customized for. I have ioDrive drivers version 1.1.1 and 1.2.1 (latter is an actual ebuild too) and hopefully soon a 1.2.4 driver. The engineer I've been dealing with at Fusion has recently been reassigned, so I now have someone else working with me on this.

If you are interested, I'd be happy to share the driver I have, but it will likely only work with the kernel configuration we use, but might be tweakable.

Let me know, the cards have been working phenomenally for us.

Treo

----------

## hvengel

Is the driver a source code driver or is it supplied as a binary blob?  Have they made any noises about submitting code to kernel.org so that the driver is part of the stock kernel?

----------

## Treo

It's a source driver, however, from what I understand, it's under a proprietary licensing agreement. I know it would be a lot better if they made it open source, but I don't think that'll happen... It's actually more than just a driver, its a set of management tools and a grooming process that has to continually run due to the nature of NAND.

I would love to see a native kernel driver, I don't like to use kernel modules for our appliances, but the driver really isn't good enough yet to be compiled in, it's crashed a few times so far.

----------

## hvengel

 *Treo wrote:*   

> It's a source driver, however, from what I understand, it's under a proprietary licensing agreement. I know it would be a lot better if they made it open source, but I don't think that'll happen... It's actually more than just a driver, its a set of management tools and a grooming process that has to continually run due to the nature of NAND.
> 
> I would love to see a native kernel driver, I don't like to use kernel modules for our appliances, but the driver really isn't good enough yet to be compiled in, it's crashed a few times so far.

 

If it is going to remain a proprietary driver then it will never be part of the kernel and it will never be possible to use one of these as a boot device.  Since I have seen press releases from the company saying that they intend to make these devices bootable then at some point the driver will need to be part of the kernel and the code will have to be open for that to happen.  Hopefully the current closed nature of the code is temporary and they will open it once the code is in a more stable state.

This device is so new that I am not in the least bit surprised that the driver is not stable yet.

----------

## Treo

Yes, this driver has sure been "fun"... I know there are plans to make it a generic driver that will work with a wide flavour of kernels, as it stands right now, they literally build a custom one for me every time they release a new version and I ask for an update (company I work for bought / buys enough of these cards to justify an engineer on their side spending that kind of time).

The card is very interesting to work with, it's performance is unbelievable. Also, the card is actually bigger than the advertised size, a 80GB card really has 99GB worth of raw chips, but they artificially reduce the size for automatic replacement of failed NAND parcels, quite impressive.

We currently churn about 1 - 2 TB of data per card per day, wear has been good at 99.5% good on average, the cards have been running for about 3 months now.

Supposedly the 1.2.4 driver and firmware bring another performance boost, which is good, we beat the cards around quite a bit as it is.

Worst thing I've found so far, a firmware upgrade required a cold boot prior to 1.2.4, which was very annoying if you have customers with remote datacenters. Only other issue is the occasional unclean detach, which causes the drive to corrupt and requires manual intervention, supposedly, 1.2.4 fixes a lot of that as well.

If there is interest, I'll keep you posted, technically, the gentoo driver we have is available for anyone to play with, provided you are running 2.6.22-gentoo-r5 and have a similar kernel conf than what we do (we run these cards on Dell PE 1950 and 2950's).

Treo

----------

