# SATA port multiplier performance

## eccerr0r

Curious what you guys are seeing with SATA port multipliers.

I had a port multiplier card and finally found a SATA card (sata_sil24) that works with port multipliers that has a regular sata connector, since I can't find my eSATA cable (ugh.  wish my ICH* worked, but nope!).  The SATA card is a PCIE x1 card...which is surely some sort of bottleneck.  It supposedly is a SATA 3.0Gb port.

Anyway I tried running an old RAID6 I set up with discrete SATA ports and got about 120MB/sec sequential reads on a hodgepodge of 500GB rust spinners on the MD device.  Eh.. not great but okay...

Currently reformatting RAID5 to see if it's any different...

What kinds of speeds should I expect, I might have to reconfigure to RAID0 to see what it does?  Should I be getting near the limit of PCIe x1 of ~250MB/sec?  I suspect the jmb363(AHCI) onboard sata controllers I have (with eSATA connector that I don't have a cable for) are also attached x1 and likely similar in performance to the sata_sil24 ...

----------

## dmpogo

 *eccerr0r wrote:*   

> Curious what you guys are seeing with SATA port multipliers.
> 
> I had a port multiplier card and finally found a SATA card (sata_sil24) that works with port multipliers that has a regular sata connector, since I can't find my eSATA cable (ugh.  wish my ICH* worked, but nope!).  The SATA card is a PCIE x1 card...which is surely some sort of bottleneck.  It supposedly is a SATA 3.0Gb port.
> 
> Anyway I tried running an old RAID6 I set up with discrete SATA ports and got about 120MB/sec sequential reads on a hodgepodge of 500GB rust spinners on the MD device.  Eh.. not great but okay...
> ...

 

I tried my back up drives which sit on esata cable, attached to internal header of ICH9 (if I am not mistaken),  and it  gives 149 MB/sec reads, but alas, it is RAID1, so it is harddrive limited performance, so this is not an answer.

----------

## eccerr0r

You have an ICH9R or something like that?  I thought ICH9's and ICH10s (non-R) do not support port multipliers.  My ICH10 I recall being able to access multiple 2T disks at the same time at full rate (> ~120MB/sec?) and sequential RAID read rate is close to the aggregate bandwidth of the disks.  Unfortunately they want to have nothing to do with the port multiplier as I don't have an ICH10R.

The individual disk performance of the 500G disks I have are nothing to write home about, in fact, dismal compared to 149MB/sec - I think some are from 70MB/sec to just about 100MB/sec a piece (mix of WD's and Hitachis).  Wondering what I should expect through the port multiplier to a SATA 3Gb port on an PCIe x1...

Yeah I think this array will ultimately be part of my backup solution -- these 500G disks are not enough to backup my 2T disks (which are RAID5'ed) but at least part of the array can be backed up.

----------

## dmpogo

 *eccerr0r wrote:*   

> You have an ICH9R or something like that?  I thought ICH9's and ICH10s (non-R) do not support port multipliers.  My ICH10 I recall being able to access multiple 2T disks at the same time at full rate (> ~120MB/sec?) and sequential RAID read rate is close to the aggregate bandwidth of the disks.  Unfortunately they want to have nothing to do with the port multiplier as I don't have an ICH10R.
> 
> The individual disk performance of the 500G disks I have are nothing to write home about, in fact, dismal compared to 149MB/sec - I think some are from 70MB/sec to just about 100MB/sec a piece (mix of WD's and Hitachis).  Wondering what I should expect through the port multiplier to a SATA 3Gb port on an PCIe x1...
> 
> Yeah I think this array will ultimately be part of my backup solution -- these 500G disks are not enough to backup my 2T disks (which are RAID5'ed) but at least part of the array can be backed up.

 

ICH9R indeed as far as I see from lspci

```

00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)

```

but the manual for my motherboard,  if I am not mistaken with the model,   Gigabyte GA-EX58-UD5P/GA-EX58-UD5,  shows ICH10R.   But maybe the manual (downloaded recently) shows some later revision or something

----------

## eccerr0r

Mind trying hdparm both disks at the same time and take the aggregate?  I wonder what that would do with the interface...

I took my system apart as it's physically very unstable.  Need to build an enclosure that will safely house the disks before I can do some more experimentation with it, as it stands it's too easy to jar a disk, which could cause damage to both disks...

----------

## dmpogo

 *eccerr0r wrote:*   

> Mind trying hdparm both disks at the same time and take the aggregate?  I wonder what that would do with the interface...
> 
> I took my system apart as it's physically very unstable.  Need to build an enclosure that will safely house the disks before I can do some more experimentation with it, as it stands it's too easy to jar a disk, which could cause damage to both disks...

 

Unfortunately, I have hardware RAID-1 on external disks in enclosure, so I do not see them separately, it comes to me as a single disk.

----------

## eccerr0r

Oh... then that's not a port multiplier then (versus a hardware RAID box)?  Perhaps it has PM versus RAID mode?

Port multipliers usually expose all disks to the host directly, or at least the one I have is JBOD and depends on md, lvm, or btrfs to deal with them as a RAID.

----------

## NeddySeagoon

eccerr0r,

Port multipliers time division multiplex the drives to the single SATA port so that the host can't tell.

Speed wise, you get whatever the SATA port is capable of as its uninformed of which drive the transfer is actually to/from.

Depending on the drives, you may or may not see a speed improvement overall.

Raid on a SATA port multiplier is scary, since each drive will be written sequentially.

Maybe if you also use a UPS?

----------

## eccerr0r

Well that's what one would expect that it's TDM the multiple drives, but there is dead time between packets due to service latency and this could be used to service another drive, so wondering how that would end up panning out increasing potential bandwidth.  Unless the multiplexing forces one to wait out latency (and disables NCQ?), then having a host bandwidth higher than disk bandwidth makes no sense ... but don't know about the actual implementation.  

Yeah it would indeed need to be sequential and thus kind of risky for RAID, though it's the same way for any RAID setup that loses power.  Speaking of which, old SCSI buses necessarily also wrote sequentially too, so could it be any worse?

Incidentally while mdadm --create the RAID5, during the resync my computer is also showing ~130MB/sec bandwidth over the 5 disks, and the sync is happening at 25MB/sec... so something really is bottlenecked.  I suspect that every 25MB/sec written parity to one disk, it needs to read 100MB/sec data from the other four disks.  This is clearly pretty bad in terms of using this for RAID5 resync, the direct to motherboard link is much faster as it can read and write each port at full speed at the same time.

I wonder if it's the PM, disks, or a combination that's the specific bottleneck.

----------

## eccerr0r

Speaking of these things, I've noticed my silicon image PCIe x1 SATA host adapter automatically detects a hot plugged SATA/port multiplier, but not sure how it's supposed to deal with a hot unplug.

When I want to hot unplug, I unmount and disable LVM on the volumes.  Then I pull the plug - but if I lsscsi the drives still show up(!)  Something seems to be wrong with hot unplug?  Hot plug works beautifully.

----------

## steve_v

 *eccerr0r wrote:*   

> Curious what you guys are seeing with SATA port multipliers.

 

The first and last time I tried one, performance was awful, compatibility was sketchy, and reliability was non-existent.

If you need more SATA ports, save yourself the pain and just go buy a half-decent PCIE HBA.

 *eccerr0r wrote:*   

> The SATA card is a PCIE x1 card...which is surely some sort of bottleneck.

 Indeed. Expect various levels of disappointing performance, relative to which generation PCIE you're talking about and how many drives are sharing that bandwidth.

Unless one intends to run mechanical disks from 2004, I personally consider anything less than 1 PCIE lane per 2 drives a potential problem. 

That's just me of course, and I'm kinda fussy... But any non-shoebox motherboard will likely have a free x8 or x16 slot for that second GPU you probably don't have, so why not put it to better use?

I went there once, doing the cheap PCIEx1 SATA cards (with and without multipliers) thing. Frankly I'd not wish the experience on anyone.

Then I bought a used IBM ServeRAID M1015 (AKA LSI SAS2008) and a SATA breakout cable for 3/5 of bugger all, flashed it to IT mode, and everything was right with the world. I'm still running it ~10 years later and it's still just lovely. I can vouch for the advertised ~350MB/s/port too.

For other recommendations, this list is very much still relevant, or hit up the likes of servethehome/freenas/unraid etc. forums.

----------

## eccerr0r

The SATA drives were meant to be offline external backup storage, and speed isn't too big an issue.  It actually isn't too bad for single drives it seems, getting close to full performance.  I was curious of multiple drive performance which is up there but not quite what I expect.  Still not horrible, but didn't quite reach theoretical limits of what I'd think should be possible (that RAID5 resync over SATA/PM I thought should have occurred about 2x as fast as it did, wasn't expecting miracles but it seemed a little low).

Just wanted to reduce number of wires to connect to the drives to deal with the offline nature.  Two is good, 6 is not.

----------

## steve_v

 *eccerr0r wrote:*   

> 
> 
> I was curious of multiple drive performance which is up there but not quite what I expect.

 

If your HBA and PM both support FIS, performance won't be too horrible... You'll still suffer on things that need concurrent access to multiple drives of course - like RAID resyncs.

Of bigger concern to me running RAID over a port multiplier is the behaviour under fault conditions. If one drive locks up the bus, it'll lock up the bus for all the drives sharing that port and down goes your array. It's even worse if the drives don't support TLER, in which case individual drive failure of any kind, even just a gnarly URE, is likely to lock up the whole shebang long enough to take the array down.

What you get running RAID over port multipliers is AID, you've created a great big SPOF and your "R" goes down the crapper.

 *eccerr0r wrote:*   

> Just wanted to reduce number of wires

 

That's why I like those mini-SAS connectors you find on real HBAs - one cable becomes 4 full-bandwidth SATA ports (that and SAS expanders do what port multiplers do, with fewer issues). You'd still need 3 cables for 6 drives though.

----------

## pjp

 *steve_v wrote:*   

> I bought a used IBM ServeRAID M1015 (AKA LSI SAS2008) and a SATA breakout cable for 3/5 of bugger all, flashed it to IT mode, and everything was right with the world. I'm still running it ~10 years later and it's still just lovely. I can vouch for the advertised ~350MB/s/port too.

  Does it require megaraid? I've never encountered a worse CLI interface than that.

----------

## steve_v

 *pjp wrote:*   

> Does it require megaraid?

 Only if you want to run it as a hardware RAID card, which you almost certainly don't. Once flashed to IT mode it's just a dumb HBA with no special tools required, and appears to the OS as JBOD. If you don't need to boot from it you can even nuke the RAID oprom entirely and shave a few seconds off boot time.  :Smile: 

The LSI CLI tools are indeed horrible, thankfully you only need to use them once, if at all. 

Oh, and on the topic of

 *eccerr0r wrote:*   

> external backup storage

 

IIRC there are SAS2008 based cards around with external mini-SAS port(s) too, for extra convenience and minimal cabling to your external cage.

Many of those cages even have SAS expander backplanes, and since SAS was actually developed with this in mind (as opposed to (s)ATA multiplexing hacks), you can happily hang 8 spinners off a single SAS cable and still get a reliable ~175MB/s per drive.  :Very Happy: 

----------

## eccerr0r

For free? :-p

Just working with items I happened to have, for curiosity sake.  No sense in buying more equipment, not like I manage a datacenter ... err.. well I do for my home network, not for a megacorp paying me to manage a datacenter...

----------

## pjp

 *steve_v wrote:*   

>  *pjp wrote:*   Does it require megaraid? Only if you want to run it as a hardware RAID card, which you almost certainly don't. Once flashed to IT mode it's just a dumb HBA with no special tools required, and appears to the OS as JBOD. If you don't need to boot from it you can even nuke the RAID oprom entirely and shave a few seconds off boot time. :)

  Good to know, thanks. Sourcing reasonably priced parts from reputable sources seems to continue to be a problem. Some day.

----------

## eccerr0r

I just noticed another anomaly with the port multiplier:

The first disk of the PM set has multcount=16.

All other disks have multcount=0 (!)

Also the first disk seems to hit 120MB/sec, others hover around 80MB/sec -- though it may be the actual drives and not the setting of multcount, so I'm not sure if this has any significance or not as libata drives don't like being hdparm -m'ed anyway...

----------

