# [SOLVED] PCI-Express throughput

## Vieri

Hi,

I'd like to know if my PCI card is working at max of its capabilities as far as throughput is concerned.

I can only connect to the system with ssh.

Dmidecode tells me that the card is connected to an "x16 PCI Express" slot.

lspci shows this:

```
07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 64 bytes

        Interrupt: pin D routed to IRQ 65

        Region 0: Memory at fdd80000 (32-bit, non-prefetchable) [size=512K]

        Region 2: I/O ports at 9000 [size=32]

        Region 3: Memory at fe100000 (32-bit, non-prefetchable) [size=16K]

        Expansion ROM at fdd00000 [disabled] [size=512K]

        Capabilities: [40] Power Management version 3

                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-

        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

                Address: 0000000000000000  Data: 0000

                Masking: 00000000  Pending: 00000000

        Capabilities: [70] MSI-X: Enable+ Count=10 Masked-

                Vector table: BAR=3 offset=00000000

                PBA: BAR=3 offset=00002000

        Capabilities: [a0] Express (v2) Endpoint, MSI 00

                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W

                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+

                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-

                        MaxPayload 128 bytes, MaxReadReq 512 bytes

                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-

                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us

                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported

                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

        Capabilities: [100 v2] Advanced Error Reporting

                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

        Capabilities: [140 v1] Device Serial Number e8-ea-6a-ff-ff-0c-4c-1c

        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

                ARICap: MFVC- ACS-, Next Function: 0

                ARICtl: MFVC- ACS-, Function Group: 0

        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

                IOVCap: Migration-, Interrupt Message Number: 000

                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-

                IOVSta: Migration-

                Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 03

                VF offset: 384, stride: 4, Device ID: 1520

                Supported Page Size: 00000553, System Page Size: 00000001

                Region 0: Memory at 0000000000000000 (64-bit, prefetchable)

                Region 3: Memory at 0000000000000000 (64-bit, prefetchable)

                VF Migration: offset: 00000000, BIR: 0

        Capabilities: [1a0 v1] Transaction Processing Hints

                Device specific mode supported

                Steering table in TPH capability structure

        Capabilities: [1d0 v1] Access Control Services

                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

        Kernel driver in use: igb
```

What does "Width x4" mean in "LnkSta: Speed 5GT/s, Width x4"?

Does this mean I have a PCI-Express 2.0 slot, and that the card has a throughput of 2.00 GB/s max.?

ThanksLast edited by Vieri on Mon Oct 21, 2019 9:23 pm; edited 1 time in total

----------

## NeddySeagoon

Vieri,

The speed on the ethernet wire is 1GBit/sec

The speed on the PCIe interface is not a problem. Both are full duplex and the bottleneck is the ethernet wire speed.  

I suspect that your 

```
07:00.3 Ethernet controller
```

is a multiport device (at least four) ethernet ports as you are showing us subfunction 3.

LnkCap is the card reporting its capabilities

LnkSta is the card reperting how its working now.

```
LnkSta: Speed 5GT/s, Width x4
```

, says its using 4 PCIe Ver 2 lanes at 5GT/s each.

----------

## Vieri

Thanks!

----------

## Ant P.

If your device isn't running at full speed, the kernel will explicitly point that out at boot:

```
Sep 13 18:48:38 [kernel] pci 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x8 link at 0000:00:02.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)
```

That's a PCIe 3.0 graphics card in a 2.0 x16 slot.

```
[  +0.000036] pci 0000:06:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:15.2 (capable of 4.000 Gb/s with 5 GT/s x1 link)
```

That example's a USB3 controller hardwired on a sketchy Zotac motherboard. Not possible to fix, but useful to figure out why it's underperforming.

----------

## Vieri

OK, I'm getting this in my log:

```
[    3.009071] igb 0000:07:00.0: eth2: (PCIe:5.0Gb/s:Width x4) e8:ea:6a:0c:4c:1c

[    3.108850] igb 0000:07:00.1: eth3: (PCIe:5.0Gb/s:Width x4) e8:ea:6a:0c:4c:1d

[    3.208868] igb 0000:07:00.2: eth4: (PCIe:5.0Gb/s:Width x4) e8:ea:6a:0c:4c:1e

[    3.308894] igb 0000:07:00.3: eth5: (PCIe:5.0Gb/s:Width x4) e8:ea:6a:0c:4c:1f

```

----------

## NeddySeagoon

Vieri,

At face value, your 4 port Ethernet card has a single PCIe per port.

That's plenty of bandwidth. As I said, the bottleneck is the Ethernet link itself.

I say "at face value" because a single PCIe:5.0Gb/s lane can keep 4  1G Ethernet ports busy but that's harder to implement on the PCIe bus end.

If you could put that card into a physical 4 lane slot that had only one lane wired, you may not notice the difference.

In theory, its enough bandwidth to keep all 4 Ethernet ports at their full data rate.

Putting what is in effect, four single lane Ethernet cards on one physical card is easier to design though.

----------

