# AsMedia ASM1061 Port Multiplier issues

## pakjebakmeel

Hi All,

I'm running an Intel D510MO mainboard with has a MiniPCIe slot. According to specs it has a PCIe 1x v2 bus capable of 500MB/s. Currently it has a Silicon Image 3132 based 2-port SATA controller installed, the second SATA port is attached to an Addonics SATA port multiplier. The PM is Sil3726 based. There are 5 3TB WD30EZRX disks attached to the PM stuffed in an mdadm raid5 array.

This works fine but I am not able to get more throughput than approx 130MB/s through the Sil3132. It appears that despite being SATA300 the card/chip has an internal limit that is bottlenecking my disk IO.

After doing some research I found that the Asmedia 1061 chip is very cheap and it is capable of reaching speeds well over 300MB/s. Because I'm running a SATA300 port multiplier I don't expect to make those speeds but at least well over 130MB/s would be nice, 250MB/s would be more in line with my expectation.

I went ahead and got myself one of these: http://www.delock.de/produkte/F_426_Module_95225/merkmale.html

2 port MiniPCIe based SATA controller that uses an Asmedia 1061 chip. Plugged it in and the BIOS shows the card is in AHCI mode, as I already had AHCI SATA enabled in the kernel it was picked up just fine:

```

[    0.622850] ahci 0000:02:00.0: irq 45 for MSI/MSI-X

[    0.622924] ahci: SSS flag set, parallel bus scan disabled

[    0.623107] ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6 Gbps 0x3 impl SATA mode

[    0.623305] ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo pmp pio slum part ccc sxs

[    0.624803] scsi4 : ahci

[    0.625285] scsi5 : ahci

[    0.625691] ata5: SATA max UDMA/133 abar m512@0xf0200000 port 0xf0200100 irq 45

[    0.625889] ata6: SATA max UDMA/133 abar m512@0xf0200000 port 0xf0200180 irq 45

```

It finds the disks through the port multiplier:

```

[    1.948274] ata5.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)

[    1.948438] ata5.01: hard resetting link

[    2.440248] ata5.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

[    2.440408] ata5.02: hard resetting link

[    2.932252] ata5.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

[    2.932410] ata5.03: hard resetting link

[    3.424248] ata5.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

[    3.424407] ata5.04: hard resetting link

[    3.916245] ata5.04: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

[    3.916403] ata5.05: hard resetting link

[    7.236028] ata5.15: qc timeout (cmd 0xe4)

[    7.236154] ata5.05: failed to read SCR 1 (Emask=0x4)

[    7.236269] ata5.05: failed to read SCR 0 (Emask=0x40)

[    7.237029] ata5.00: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133

[    7.237148] ata5.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

[    7.237910] ata5.00: configured for UDMA/133

[    7.238611] ata5.01: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133

[    7.238732] ata5.01: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

[    7.239484] ata5.01: configured for UDMA/133

[    7.240187] ata5.02: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133

[    7.240312] ata5.02: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

[    7.241070] ata5.02: configured for UDMA/133

[    7.241771] ata5.03: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133

[    7.241892] ata5.03: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

[    7.242648] ata5.03: configured for UDMA/133

[    7.243350] ata5.04: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133

[    7.243472] ata5.04: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

[    7.244241] ata5.04: configured for UDMA/133

[    7.244429] ata5: EH complete

```

And after booting the array was properly assembled. "Benchmarking" individual disks looks fine:

```

/dev/sdc:

 Timing cached reads:   1574 MB in  2.00 seconds = 787.55 MB/sec

 Timing buffered disk reads: 438 MB in  3.01 seconds = 145.53 MB/sec

/dev/sdd:

 Timing cached reads:   1540 MB in  2.00 seconds = 770.47 MB/sec

 Timing buffered disk reads: 476 MB in  3.00 seconds = 158.51 MB/sec

/dev/sde:

 Timing cached reads:   1540 MB in  2.00 seconds = 770.61 MB/sec

 Timing buffered disk reads: 426 MB in  3.02 seconds = 141.08 MB/sec

/dev/sdf:

 Timing cached reads:   1544 MB in  2.00 seconds = 772.48 MB/sec

 Timing buffered disk reads: 424 MB in  3.02 seconds = 140.44 MB/sec

/dev/sdg:

 Timing cached reads:   1548 MB in  2.00 seconds = 773.76 MB/sec

 Timing buffered disk reads: 428 MB in  3.01 seconds = 142.16 MB/sec

```

The real problem occurs when accessing the array:

```

/dev/md5:

 Timing cached reads:   1428 MB in  2.00 seconds = 714.01 MB/sec

 Timing buffered disk reads:  70 MB in  3.34 seconds =  20.93 MB/sec

```

In comparison; the Sil3132 does this (while rebuilding actually):

```

/dev/md5:

 Timing cached reads:   1258 MB in  2.00 seconds = 629.35 MB/sec

 Timing buffered disk reads: 358 MB in  3.01 seconds = 118.80 MB/sec

```

Stumped as I was I ran the hdparm a couple of times to make sure the result was consistent. The results were even worse; sometimes 2MB/s up to 7MB/s. Hmm weird; so to confirm I wrote a 1GiB test file on the array which was very slow. After a minute or so it stalled and the array kinda crashed. When looking at the mountpoint it was showing files with garbled names and messed up user/group owners.

mdadm --detail /dev/md5 showed that 3 disks were in a spare state? Like this: /dev/sdc1(S). Stopped and started the array, no go. Mdadm examine showed the disks were 'clean' so I force assembled and it did, just one disk missing. Kicked it out of the array, wiped superblock and put it back in, all looks ok now and it's rebuilding. Back on the old Sil3132 controller ofcourse.

So after having quite a scare my data might have been lost permantly;

- The asm1061 works fine on individual disks

- The asm1061 can see all disks behind the PM

- Performance on the disks behind the PM is fine

- When accessing all disks at once by generating IO to /dev/md5 it crawls to a halt

- The sil3132 works fine but just limited to 130MB/s

smartctl looks fine for each disk:

/dev/sdc

```

SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

  3 Spin_Up_Time            0x0027   174   174   021    Pre-fail  Always       -       6283

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       34

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       444

 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0

 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0

 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       34

192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       28

193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       114

194 Temperature_Celsius     0x0022   117   115   000    Old_age   Always       -       33

196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

```

/dev/sdd

```

SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

  3 Spin_Up_Time            0x0027   174   173   021    Pre-fail  Always       -       6300

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       12

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       204

 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0

 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0

 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       12

192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7

193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       52

194 Temperature_Celsius     0x0022   121   121   000    Old_age   Always       -       29

196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

```

/dev/sde

```

SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

  3 Spin_Up_Time            0x0027   177   177   021    Pre-fail  Always       -       6116

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       447

 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0

 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0

 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       35

192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       29

193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       144

194 Temperature_Celsius     0x0022   120   118   000    Old_age   Always       -       30

196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

```

/dev/sdf

```

SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

  3 Spin_Up_Time            0x0027   179   179   021    Pre-fail  Always       -       6008

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       447

 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0

 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0

 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       35

192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       29

193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       116

194 Temperature_Celsius     0x0022   118   116   000    Old_age   Always       -       32

196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

```

/dev/sdg

```

SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

  3 Spin_Up_Time            0x0027   182   181   021    Pre-fail  Always       -       5883

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       447

 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0

 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0

 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       35

192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       29

193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       112

194 Temperature_Celsius     0x0022   116   114   000    Old_age   Always       -       34

196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

```

Anyone any clue? hints?   :Rolling Eyes: 

----------

## s4e8

asm chip don't support pmp fis base switch, it's non-capable for raid. 

sil3132 has fbs support, but limited throughput per port.

I recommed buying a marvell 91xx base adapter.

----------

## pakjebakmeel

 *s4e8 wrote:*   

> asm chip don't support pmp fis base switch, it's non-capable for raid. 
> 
> sil3132 has fbs support, but limited throughput per port.
> 
> I recommed buying a marvell 91xx base adapter.

 

Does it? On all spec pages it says that asm1061 is PMP compatible, also it can see all the drives behind the PMP and I have seen successes with drive cages on an asm1061. Would it not just only see 1 drive if it cannot handle FIS PMP?

```

Engaged in High Speed I/O solution development, Asmedia Technology is committed to enlarging product portfolio with introducing PCI Express Products. The ASM1061, x1 PCI Express to two ports of Serial ATA, enables Serial ATA PHY up to 6Gbps high speed interface, following Serial ATA Revision 3.0 specification.

General Features

- Option Rom support through 64K Byte SPI flash

- 20MHz external crystal

- Integrated 3.3V to 1.2V switch regulator

- 3.3/1.2V Power Supply

- Industry Specifications Compliance:

   PCI Express Base Specification Rev. 2.0

   Bus Master Programming Interface to IDE ATA Controller Rev. 1.0

   PCI Express Card Electromechanical Rev . 2.0

   Serial ATA AHCI Spec. Rev.1.3

   Serial ATA Revision 3.0

 

PCI Express Features

- x1 PCI Express lane for 2.5 and 5GHz signaling

- Single virtual channel

- SSC support

- ECRC and Advanced Error Reporting capability

- 100MHz differential PCI Express reference clock in

- Maximum Payload up to 128 bytes

Serial ATA Features

- 2 ports Serial ATA PHY for 1.5, 3.0 and 6.0GHz signaling

- Support Native Command Queue

- Support Gen 1m and 2m SATA PHY

- Support Port Multiplier
```

I've looked around the web and found some other cards similar to this one that use the ASM1061 and they all state the controller is FIS-Based switching compatible.. Not found any whitepaper or datasheet from AsMedia yet.

This one for example: http://www.mediasonic.ca/product.php?id=1357626065

```

ASMEDIA-asm1061 chipset

Support Port Multiplier and FIS-Based switch

```

----------

## s4e8

ASM1061 has no FBS support, your dmesg say it.

After do some testing with Marvell 9172. The linux ahci driver don't detect disks at all. The windows does see disks but the performance is poor.

Sil3132 is the cheapest working PMP adapter.

Another adapter has best sata pmp support is highpoint 272x SAS2 adapter(marvell 9485), w/ proprietary driver. The mvsas or other libsas based driver or all open source linux sas drivers has no sata pmp at all.

Add some info. After force kernel to ignore the fbs flag, Marvell 9172 work fine with the sata pmp. So the chip doesn't support fbs.

----------

## pakjebakmeel

 *s4e8 wrote:*   

> ASM1061 has no FBS support, your dmesg say it.
> 
> After do some testing with Marvell 9172. The linux ahci driver don't detect disks at all. The windows does see disks but the performance is poor.
> 
> Sil3132 is the cheapest working PMP adapter.
> ...

 

My understanding is that "FIS Based Switching" implies that the host controller is able to "tag" individual packets/blocks/actions (or whatever goes down a SATA wire) with the "label" of the destination disk. The port multiplier uses this "tag" to route the packet to the correct disk. Without this ability the host controller will only detect 1 disk, namely the one connected to the first most port of the multiplier. (As described in the manuals of the PM's)

Fact is that when the server boots the bios shows 5 connected drives, even better; booted into Gentoo I get 5 device nodes and I can benchmark every individual device. My conclusion would be that the controller supports FIS Based Switching. Else it would only 'see' disk number 1 behind the multiplier. It's only when you mount the MD array and generate traffic over all disk that things go belly up. I would say the kernel's ahci support for this controller is bugged/flaky rather then the controller not supporting FIS Based Switching and this needs patching.

Doesn't the fact that during boot 5 drives are visible and the fact that I can see and benchmark 5 different device nodes prove that the controller is capable of FBS?

I'm not arguing but looking for an explanation on why this isn't working. Any who can explain this behaviour to me?  :Question: 

----------

## s4e8

Command Based switch means You can switch disk by command, like flip papers. When you access disk1, you can't access disk2 concurrently. This mode just like the primary and slave disk on same PATA cable.

FIS is a request based switch, sth like the SCSI protocol.

----------

## pakjebakmeel

Ahhhh that makes sense. Thank you for this explanation.

It works fine when accessing individual disks and it detects all of them.. Benchmarking several makes it slow and when benchmarking the array things go bad. Probably MDADM doesn't like the delays and eventually fails a disk?

Bit disappointed as most websites advertise the controller as FIS capable.   :Evil or Very Mad: 

----------

## ciscoy2g15

I know this thread is a bit old.  But perhaps someone may find it useful.  After having several RAID arrays get corrupted, and finally getting familiar with mdadm to be able to recover from these resets, I found this:

http://www.linuxquestions.org/questions/linux-hardware-18/sata-libata-link-reset-problem-with-port-multiplier-pmp-sil3132-sil3726-4175445070/

It looks like you can add the libata-pmp.ko kernel module so that you can prevent these resets.  I haven't been successful in doing this for Ubuntu, and was hoping Gentoo already had it loaded by default, but wasn't the case.  I guess I'll have to do a bit more reading about how to create my own kernel module based on this:

http://lxr.free-electrons.com/source/drivers/ata/libata-pmp.c

It seems that it's also dependent on a couple of other files, which you can find here (one directory up):

http://lxr.free-electrons.com/source/drivers/ata/

But kept getting problems when compiling the kernel module, stating that the symbols were incorrect or something to that effect (I'm away from that system, and don't want to bring it down to test at this time).  At any rate, I just wanted to post this on here.

If anyone has any pointers as to how to go about compiling this kernel in either Gentoo or Ubuntu, it would be helpful.  Otherwise, I'll post an update once I figure it out.

Peace

----------

