# Unable to fully initialize a Chelsio T420-BT PCIe adapter

## blinky

First off I'm quite new to Gentoo.

I have a Chelsio T420-BT that doesn't seem to want to initialize. The kernel reports that the physical/virtual functions (SR-IOV) appear to initialize but the physical NIC itself doesn't. There were some kernel warnings that I don't understand:

Kernel log:

full kernel log here

0000:01:00.4 is the physical NIC in question.

0000:01:00.0 -> 3 are the 'physical functions'

```

[  +0.000091] pci 0000:01:00.0: [1425:4009] type 00 class 0x020000

[  +0.000019] pci 0000:01:00.0: reg 0x10: [mem 0xf7380000-0xf73bffff 64bit]

[  +0.000016] pci 0000:01:00.0: reg 0x20: [mem 0xf73cc000-0xf73cdfff 64bit]

[  +0.000006] pci 0000:01:00.0: reg 0x30: [mem 0xf7180000-0xf71fffff pref]

[  +0.000043] pci 0000:01:00.0: reg 0x1e8: [mem 0x00000000-0x00000fff 64bit]

[  +0.000002] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x0000ffff 64bit] (contains BAR0 for 16 VFs)

[  +0.000138] pci 0000:01:00.0: reg 0x1f8: [mem 0x00000000-0x00001fff 64bit]

[  +0.000002] pci 0000:01:00.0: VF(n) BAR4 space: [mem 0x00000000-0x0001ffff 64bit] (contains BAR4 for 16 VFs)

[  +0.000159] pci 0000:01:00.0: System wakeup disabled by ACPI

[  +0.000116] pci 0000:01:00.1: [1425:4009] type 00 class 0x020000

[  +0.000019] pci 0000:01:00.1: reg 0x10: [mem 0xf7340000-0xf737ffff 64bit]

[  +0.000016] pci 0000:01:00.1: reg 0x20: [mem 0xf73ca000-0xf73cbfff 64bit]

[  +0.000006] pci 0000:01:00.1: reg 0x30: [mem 0xf7100000-0xf717ffff pref]

[  +0.000038] pci 0000:01:00.1: reg 0x168: [mem 0x00000000-0x00000fff 64bit]

[  +0.000002] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0x00000000-0x0000ffff 64bit] (contains BAR0 for 16 VFs)

[  +0.000141] pci 0000:01:00.1: reg 0x178: [mem 0x00000000-0x00001fff 64bit]

[  +0.000001] pci 0000:01:00.1: VF(n) BAR4 space: [mem 0x00000000-0x0001ffff 64bit] (contains BAR4 for 16 VFs)

[  +0.000185] pci 0000:01:00.2: [1425:4009] type 00 class 0x020000

[  +0.000018] pci 0000:01:00.2: reg 0x10: [mem 0xf7300000-0xf733ffff 64bit]

[  +0.000017] pci 0000:01:00.2: reg 0x20: [mem 0xf73c8000-0xf73c9fff 64bit]

[  +0.000043] pci 0000:01:00.2: reg 0x168: [mem 0x00000000-0x00000fff 64bit]

[  +0.000001] pci 0000:01:00.2: VF(n) BAR0 space: [mem 0x00000000-0x0000ffff 64bit] (contains BAR0 for 16 VFs)

[  +0.000140] pci 0000:01:00.2: reg 0x178: [mem 0x00000000-0x00001fff 64bit]

[  +0.000001] pci 0000:01:00.2: VF(n) BAR4 space: [mem 0x00000000-0x0001ffff 64bit] (contains BAR4 for 16 VFs)

[  +0.000174] pci 0000:01:00.3: [1425:4009] type 00 class 0x020000

[  +0.000018] pci 0000:01:00.3: reg 0x10: [mem 0xf72c0000-0xf72fffff 64bit]

[  +0.000017] pci 0000:01:00.3: reg 0x20: [mem 0xf73c6000-0xf73c7fff 64bit]

[  +0.000042] pci 0000:01:00.3: reg 0x168: [mem 0x00000000-0x00000fff 64bit]

[  +0.000001] pci 0000:01:00.3: VF(n) BAR0 space: [mem 0x00000000-0x0000ffff 64bit] (contains BAR0 for 16 VFs)

[  +0.000139] pci 0000:01:00.3: reg 0x178: [mem 0x00000000-0x00001fff 64bit]

[  +0.000002] pci 0000:01:00.3: VF(n) BAR4 space: [mem 0x00000000-0x0001ffff 64bit] (contains BAR4 for 16 VFs)

[  +0.000173] pci 0000:01:00.4: [1425:4409] type 00 class 0x020000

[  +0.000018] pci 0000:01:00.4: reg 0x10: [mem 0xf7280000-0xf72bffff 64bit]

[  +0.000008] pci 0000:01:00.4: reg 0x18: [mem 0xf6800000-0xf6ffffff 64bit]

[  +0.000008] pci 0000:01:00.4: reg 0x20: [mem 0xf73c4000-0xf73c5fff 64bit]

[  +0.000062] pci 0000:01:00.5: [1425:4509] type 00 class 0x010000

[  +0.000018] pci 0000:01:00.5: reg 0x10: [mem 0xf7240000-0xf727ffff 64bit]

[  +0.000017] pci 0000:01:00.5: reg 0x20: [mem 0xf73c2000-0xf73c3fff 64bit]

[  +0.000005] pci 0000:01:00.5: reg 0x30: [mem 0xf7080000-0xf70fffff pref]

[  +0.000062] pci 0000:01:00.6: [1425:4609] type 00 class 0x0c0400

[  +0.000018] pci 0000:01:00.6: reg 0x10: [mem 0xf7200000-0xf723ffff 64bit]

[  +0.000017] pci 0000:01:00.6: reg 0x20: [mem 0xf73c0000-0xf73c1fff 64bit]

[  +0.000005] pci 0000:01:00.6: reg 0x30: [mem 0xf7000000-0xf707ffff pref]

[  +0.000058] pci 0000:01:00.7: [1425:0000] type 00 class 0x020000

[  +0.000132] pci 0000:01:00.0: BAR 11: assigned [mem 0xf73ce000-0xf73edfff 64bit]

[  +0.000127] pci 0000:01:00.1: BAR 11: no space for [mem size 0x00020000 64bit]

[  +0.000122] pci 0000:01:00.1: BAR 11: failed to assign [mem size 0x00020000 64bit]

[  +0.000119] pci 0000:01:00.2: BAR 11: no space for [mem size 0x00020000 64bit]

[  +0.000912] pci 0000:01:00.2: BAR 11: failed to assign [mem size 0x00020000 64bit]

[  +0.000113] pci 0000:01:00.3: BAR 11: no space for [mem size 0x00020000 64bit]

[  +0.000116] pci 0000:01:00.3: BAR 11: failed to assign [mem size 0x00020000 64bit]

[  +0.000131] pci 0000:01:00.0: BAR 7: assigned [mem 0xf73ee000-0xf73fdfff 64bit]

[  +0.000120] pci 0000:01:00.1: BAR 7: no space for [mem size 0x00010000 64bit]

[  +0.000091] pci 0000:01:00.1: BAR 7: failed to assign [mem size 0x00010000 64bit]

[  +0.000118] pci 0000:01:00.2: BAR 7: no space for [mem size 0x00010000 64bit]

[  +0.000095] pci 0000:01:00.2: BAR 7: failed to assign [mem size 0x00010000 64bit]

[  +0.000119] pci 0000:01:00.3: BAR 7: no space for [mem size 0x00010000 64bit]

[  +0.000082] pci 0000:01:00.3: BAR 7: failed to assign [mem size 0x00010000 64bit]

[  +0.000092] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt

[  +0.000084] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt

[  +0.000085] pci 0000:01:00.2: Signaling PME through PCIe PME interrupt

[  +0.000104] pci 0000:01:00.3: Signaling PME through PCIe PME interrupt

[  +0.000084] pci 0000:01:00.4: Signaling PME through PCIe PME interrupt

[  +0.000085] pci 0000:01:00.5: Signaling PME through PCIe PME interrupt

[  +0.000083] pci 0000:01:00.6: Signaling PME through PCIe PME interrupt

[  +0.000094] pci 0000:01:00.7: Signaling PME through PCIe PME interrupt

[  +7.602379] cxgb4 0000:01:00.4: mbox 4: 1c0002900000004 1400100001000000 0 0 0 0 0 0

[  +0.000004] cxgb4 0000:01:00.4: command 0x1 in mailbox 4 timed out

[  +0.000002] cxgb4 0000:01:00.4: Firmware reports adapter error: Crash

[  +0.000140] cxgb4: probe of 0000:01:00.4 failed with error -16

```

lspci:

lspci -vvnn here

01:00:7 is unassigned as stated in the user's guide.

```

01:00.0 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:0000]

   Kernel driver in use: cxgb4

   Kernel modules: cxgb4

01:00.1 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:0000]

   Kernel driver in use: cxgb4

   Kernel modules: cxgb4

01:00.2 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:0000]

   Kernel driver in use: cxgb4

   Kernel modules: cxgb4

01:00.3 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:0000]

   Kernel driver in use: cxgb4

   Kernel modules: cxgb4

01:00.4 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4409]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:0000]

   Kernel modules: cxgb4

01:00.5 SCSI storage controller [0100]: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:4509]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:0000]

01:00.6 Fibre Channel [0c04]: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:4609]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:0000]

01:00.7 Ethernet controller [0200]: Chelsio Communications Inc Device [1425:0000]

   Subsystem: Chelsio Communications Inc Device [1425:0000]

```

Kernel config:

full kernel config here

```

CONFIG_NET_VENDOR_CHELSIO=y

# CONFIG_CHELSIO_T1 is not set

# CONFIG_CHELSIO_T3 is not set

CONFIG_CHELSIO_T4=m

CONFIG_CHELSIO_T4VF=m

```

Firmware versions:

Sorry, I didn't know a more effective method for getting firmware versions. This is a listing of /lib64/firmware/cxgb4/ and are installed by linux-firmware. It appears to be v1.14.4

```

aq1202_fw.cld

bcm8483.bin

t4fw-1.14.4.0.bin

t4fw.bin

t5fw-1.14.4.0.bin

t5fw.bin

```

Kernel version:

```

4.4.8-hardened-r1 x86_64 GNU/Linux

```

----------

## NeddySeagoon

blinky,

Welcome to Gentoo.

Notice that 

```
01:00.5 SCSI storage controller [0100]: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:4509]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:0000]

01:00.6 Fibre Channel [0c04]: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:4609]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:0000] 
```

there are no drivers loaded for those elements of your card, so they won't be working just now.

Grepping the kernel source for your vendor ID

```
/usr/src/linux $ grep -R 1425 ./
```

produces one interesting hit is the SCSI region of the code.

```
./drivers/scsi/csiostor/csio_hw.h:#define CSIO_VENDOR_ID            0x1425
```

That's a SCSI driver that matches your vendor ID.

That file header says 

```
/*

 * This file is part of the Chelsio FCoE driver for Linux.
```

which looks promising too.

Reading the Kconfig file in that directory shows.

```
 config SCSI_CHELSIO_FCOE

        tristate "Chelsio Communications FCoE support"

        depends on PCI && SCSI

        depends on SCSI_FC_ATTRS
```

Looking in your kernel .config shows that the symbol SCSI_CHELSIO_FCOE does not appear, so you don't have that driver built.

However, we know it depends on SCSI_FC_ATTRS

```
# SCSI Transports

#

CONFIG_SCSI_SPI_ATTRS=y

# CONFIG_SCSI_FC_ATTRS is not set
```

which is off, so the option is hidden.

Rebuild your kernel, using menuconfig to enable CONFIG_SCSI_FC_ATTRS which is a menu, then inside that menu, enable SCSI_CHELSIO_FCOE.

That should fix the Fibre Channel Over Ethernet part of your card.

----------

## blinky

Hi, thanks for taking the time to show the steps you took - invaluable information.

I deliberately didn't compile the modules for those device because I didn't think I needed them. The NIC on the card is a standard ethernet RJ45 (which is how I intend to use it) and it is function #4 - the device having the issue:

full (pci bus 01) kernel log here

```

[  +0.000169] pci 0000:01:00.4: [1425:4409] type 00 class 0x020000

[  +0.000019] pci 0000:01:00.4: reg 0x10: [mem 0xf7280000-0xf72bffff 64bit]

[  +0.000007] pci 0000:01:00.4: reg 0x18: [mem 0xf6800000-0xf6ffffff 64bit]

[  +0.000008] pci 0000:01:00.4: reg 0x20: [mem 0xf73c4000-0xf73c5fff 64bit]

[  +0.000080] pci 0000:01:00.4: Signaling PME through PCIe PME interrupt

[  +7.664926] cxgb4 0000:01:00.4: mbox 4: 1c0002900000004 1400100001000000 0 0 0 0 0 0

[  +0.000003] cxgb4 0000:01:00.4: command 0x1 in mailbox 4 timed out

[  +0.000002] cxgb4 0000:01:00.4: Firmware reports adapter error: Crash

[  +0.000144] cxgb4: probe of 0000:01:00.4 failed with error -16

```

Pardon my ignorance, and I hope you can correct me here, but would the improper initialization of other functions cause the bad behaviour seen here? More succinctly: is it necessary for all the other functions to be properly working in order for any of them to work?  Function #6 is the FCOE  offload device which I have no intention of using.

However, I compiled in support for it:

```

01:00.5 SCSI storage controller [0100]: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:4509]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:0000]

01:00.6 Fibre Channel [0c04]: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:4609]

   Subsystem: Chelsio Communications Inc T420-BT Unified Wire Storage Controller [1425:0000]

   Kernel modules: csiostor

```

Although I was unable to get the iSCSI offload function working (I won't be using it): again, do you think I need this working for the whole card to function properly?

full kernel config here

```

CONFIG_SCSI_FC_ATTRS=y

# CONFIG_SCSI_CXGB3_ISCSI is not set

CONFIG_SCSI_CXGB4_ISCSI=m

CONFIG_SCSI_CHELSIO_FCOE=m

CONFIG_NET_VENDOR_CHELSIO=y

# CONFIG_CHELSIO_T1 is not set

# CONFIG_CHELSIO_T3 is not set

CONFIG_CHELSIO_T4=m

CONFIG_CHELSIO_T4_DCB=y

CONFIG_CHELSIO_T4VF=m

```

----------

## NeddySeagoon

blinky,

From your first post,

```
01:00.0 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

01:00.1 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

01:00.2 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

01:00.3 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4009]

01:00.4 Ethernet controller [0200]: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller [1425:4409]
```

Notice that function 4 has a different deviceID, 4409, compared to functions 0..3, 4009, which says that in some way or another, its a different device.   

The Chelsio website shows 1.15.37.0 as the latest firmware.

```
$ ls -l /lib/firmware/cxgb4

total 2608

-rw-r--r-- 1 root root 179202 Jul 31 12:02 aq1202_fw.cld

-rw-r--r-- 1 root root 208900 Jul 31 12:02 bcm8483.bin

-rw-r--r-- 1 root root 539648 Jul 31 12:02 t4fw-1.14.4.0.bin

-rw-r--r-- 1 root root 541184 Jul 31 12:02 t4fw-1.15.37.0.bin

lrwxrwxrwx 1 root root     18 Jul 31 12:02 t4fw.bin -> t4fw-1.15.37.0.bin

-rw-r--r-- 1 root root 593920 Jul 31 12:02 t5fw-1.14.4.0.bin

-rw-r--r-- 1 root root 596992 Jul 31 12:02 t5fw-1.15.37.0.bin

lrwxrwxrwx 1 root root     18 Jul 31 12:02 t5fw.bin -> t5fw-1.15.37.0.bin
```

which is what the testing version of linux-firmware gives me.

Its worth trying a firmware update.

I would not expect to load drivers for functions I wasn't going to use but there is a lot of poorly designed hardware out there, so I wouldn't rule it out either.

----------

## blinky

As it turns out my device is faulty - I'm ordering a replacement; I'll keep you updated.

----------

