# SOLVED - enable Virtual NIC interfaces (SR-IOV) using sysfs

## gordonp

I  have a KVM / QEMU host, with several guests.  It works very well.

For this Post, all my cut-and-pastes, and my questions, involve the HOST ONLY.

And, I am using OpenRC.

Networking *is not* bridged - rather, I have a 10Gb fiber NIC (Intel X520), and this offers up to 64 Virtual Interfaces (VIFs), allowing direct-attach to each guest.  I create 15 such VIFs, and use about 5.

To create the VIFs, I have this in my /etc/default/grub, which passes the necessary parameters to the host kernel during boot:

GRUB_CMDLINE_LINUX_DEFAULT="splash=verbose,theme:emerge-world vga=795 net.ifnames=0 intel_iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 ixgbe.max_vfs=15"

But dmesg shows me this warning; changes are indicated:

ixgbe 0000:01:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

Looking around, I find where the number of virtual-interfaces are set; directly entering values does not persist across reboots although it does work well in a running machine::

```
# find /sys -name sriov_numvfs

/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/sriov_numvfs

# cat /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/sriov_numvfs

15
```

I tried placing entries in /etc/modprobe.d/ixgbe.conf but this did not work:

```
/etc/modprobe.d # cat ixgbe.conf 

# The number of VFs that can be allocated depends on the number of traffic

# classes that can be enabled. The configurable number of traffic classes for

# each enabled VF is as follows:

# 0 - 15 VFs = Up to 8 traffic classes, depending on device support

# 16 - 31 VFs = Up to 4 traffic classes

# 32 - 63 VFs = 1 traffic class

options ixgbe max_vfs=15
```

So, the kernel command line works, but how do I move this setting to pci sysfs?  Is it something in /etc/sysctl.conf?  If "yes", what is the syntax (especially the path - my hacky attempts here did not work)?

TIA!Last edited by gordonp on Thu Jan 02, 2020 10:55 pm; edited 1 time in total

----------

## gordonp

Any thoughts or suggestions?

TIA,

----------

## GDH-gentoo

 *gordonp wrote:*   

> So, the kernel command line works, but how do I move this setting to pci sysfs?

 Probably with udev rules (see step 4 of the procedure).

----------

## gordonp

 *GDH-gentoo wrote:*   

> Probably with udev rules

 

OK... Now that you suggest this, @GDH-gentoo, I recall trying this, and it did not work.  However, I do  believe this approach may work well, so I'll re-visit it.

I created the file /etc/udev/rules.d/eth0.rules, with the single line:

```
ACTION=="add", SUBSYSTEM=="net", ENV{ID_NET_DRIVER}=="ixgbe", ATTR{device/sriov_numvfs}="15"
```

Testing this is not happy:

```
# udevadm test /etc/udev/rules.d/eth0.rules

This program is for debugging only, it does not run any program

specified by a RUN key. It may show incorrect results, because

some values may be different, or not available at a simulation run.

Load module index

Network interface NamePolicy= disabled on kernel command line, ignoring.

Parsed configuration file /lib/systemd/network/99-default.link

Created link configuration context.

Reading rules file: /lib/udev/rules.d/10-dm.rules

Reading rules file: /lib/udev/rules.d/11-dm-lvm.rules

/lib/udev/rules.d/11-dm-lvm.rules:40 Invalid value for OPTIONS key, ignoring: 'event_timeout=180'

/lib/udev/rules.d/11-dm-lvm.rules:40 The line takes no effect, ignoring.

Reading rules file: /lib/udev/rules.d/13-dm-disk.rules

Reading rules file: /lib/udev/rules.d/40-gentoo.rules

Reading rules file: /lib/udev/rules.d/50-udev-default.rules

Reading rules file: /lib/udev/rules.d/60-block.rules

Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

Reading rules file: /lib/udev/rules.d/60-drm.rules

Reading rules file: /lib/udev/rules.d/60-evdev.rules

Reading rules file: /lib/udev/rules.d/60-input-id.rules

Reading rules file: /lib/udev/rules.d/60-persistent-alsa.rules

Reading rules file: /lib/udev/rules.d/60-persistent-input.rules

Reading rules file: /lib/udev/rules.d/60-persistent-storage-tape.rules

Reading rules file: /lib/udev/rules.d/60-persistent-storage.rules

Reading rules file: /lib/udev/rules.d/60-persistent-v4l.rules

Reading rules file: /lib/udev/rules.d/60-sensor.rules

Reading rules file: /lib/udev/rules.d/60-serial.rules

Reading rules file: /run/udev/rules.d/61-dev-root-link.rules

Reading rules file: /lib/udev/rules.d/64-btrfs.rules

Reading rules file: /lib/udev/rules.d/65-kvm.rules

Reading rules file: /lib/udev/rules.d/69-dm-lvm-metad.rules

Reading rules file: /lib/udev/rules.d/70-joystick.rules

Reading rules file: /lib/udev/rules.d/70-mouse.rules

Reading rules file: /lib/udev/rules.d/70-touchpad.rules

Reading rules file: /lib/udev/rules.d/75-net-description.rules

Reading rules file: /lib/udev/rules.d/75-probe_mtd.rules

Reading rules file: /lib/udev/rules.d/78-sound-card.rules

Reading rules file: /lib/udev/rules.d/80-drivers.rules

Reading rules file: /lib/udev/rules.d/80-grsec.rules

Reading rules file: /etc/udev/rules.d/80-net-name-slot.rules

Reading rules file: /etc/udev/rules.d/80-net-setup-link.rules

Reading rules file: /lib/udev/rules.d/90-network.rules

Reading rules file: /lib/udev/rules.d/95-dm-notify.rules

Reading rules file: /lib/udev/rules.d/96-e2scrub.rules

Reading rules file: /etc/udev/rules.d/eth0.rules

Failed to open device '/sys/etc/udev/rules.d/eth0.rules': No such device

Unload module index

Unloaded link configuration context.

```

I wonder if there is some path, or something simple I am not understanding?  Can you suggest modification to my eth0.rules file to get farther along?

Thank-you!

----------

## GDH-gentoo

 *gordonp wrote:*   

> 
> 
> ```
> # udevadm test /etc/udev/rules.d/eth0.rules
> 
> ...

 udevadm should be called with the sysfs pathname of your Ethernet interface, or one of its symbolic links, not the pathname of a rules file. The command shows that the file is being picked up. If the interface is really named eth0, udevadm test /sys/class/net/eth0 should do what you wanted (check with ls that the pathname is correct).

----------

## gordonp

 *GDH-gentoo wrote:*   

> udevadm should be called with the sysfs pathname.

 

Oh, thank-you!  I now see better results from this udevadm command:

```
# udevadm test  /sys/class/net/eth0/

This program is for debugging only, it does not run any program

specified by a RUN key. It may show incorrect results, because

some values may be different, or not available at a simulation run.

Load module index

Network interface NamePolicy= disabled on kernel command line, ignoring.

Parsed configuration file /lib/systemd/network/99-default.link

Created link configuration context.

Reading rules file: /lib/udev/rules.d/10-dm.rules

Reading rules file: /lib/udev/rules.d/11-dm-lvm.rules

/lib/udev/rules.d/11-dm-lvm.rules:40 Invalid value for OPTIONS key, ignoring: 'event_timeout=180'

/lib/udev/rules.d/11-dm-lvm.rules:40 The line takes no effect, ignoring.

Reading rules file: /lib/udev/rules.d/13-dm-disk.rules

Reading rules file: /lib/udev/rules.d/40-gentoo.rules

Reading rules file: /lib/udev/rules.d/50-udev-default.rules

Reading rules file: /lib/udev/rules.d/60-block.rules

Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

Reading rules file: /lib/udev/rules.d/60-drm.rules

Reading rules file: /lib/udev/rules.d/60-evdev.rules

Reading rules file: /lib/udev/rules.d/60-input-id.rules

Reading rules file: /lib/udev/rules.d/60-persistent-alsa.rules

Reading rules file: /lib/udev/rules.d/60-persistent-input.rules

Reading rules file: /lib/udev/rules.d/60-persistent-storage-tape.rules

Reading rules file: /lib/udev/rules.d/60-persistent-storage.rules

Reading rules file: /lib/udev/rules.d/60-persistent-v4l.rules

Reading rules file: /lib/udev/rules.d/60-sensor.rules

Reading rules file: /lib/udev/rules.d/60-serial.rules

Reading rules file: /run/udev/rules.d/61-dev-root-link.rules

Reading rules file: /lib/udev/rules.d/64-btrfs.rules

Reading rules file: /lib/udev/rules.d/65-kvm.rules

Reading rules file: /lib/udev/rules.d/69-dm-lvm-metad.rules

Reading rules file: /lib/udev/rules.d/70-joystick.rules

Reading rules file: /lib/udev/rules.d/70-mouse.rules

Reading rules file: /lib/udev/rules.d/70-touchpad.rules

Reading rules file: /lib/udev/rules.d/75-net-description.rules

Reading rules file: /lib/udev/rules.d/75-probe_mtd.rules

Reading rules file: /lib/udev/rules.d/78-sound-card.rules

Reading rules file: /lib/udev/rules.d/80-drivers.rules

Reading rules file: /lib/udev/rules.d/80-grsec.rules

Reading rules file: /etc/udev/rules.d/80-net-name-slot.rules

Reading rules file: /etc/udev/rules.d/80-net-setup-link.rules

Reading rules file: /lib/udev/rules.d/90-network.rules

Reading rules file: /lib/udev/rules.d/95-dm-notify.rules

Reading rules file: /lib/udev/rules.d/96-e2scrub.rules

Reading rules file: /etc/udev/rules.d/eth0.rules

Using default interface naming scheme 'v243'.

DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0

INTERFACE=eth0

IFINDEX=2

ACTION=add

SUBSYSTEM=net

ID_NET_NAMING_SCHEME=v243

ID_NET_NAME_MAC=enx90e2ba91b430

ID_OUI_FROM_DATABASE=Intel Corporate

ID_NET_NAME_PATH=enp1s0

ID_BUS=pci

ID_VENDOR_ID=0x8086

ID_MODEL_ID=0x10fb

ID_PCI_CLASS_FROM_DATABASE=Network controller

ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller

ID_VENDOR_FROM_DATABASE=Intel Corporation

ID_MODEL_FROM_DATABASE=82599ES 10-Gigabit SFI/SFP+ Network Connection (Ethernet Server Adapter X520-1)

USEC_INITIALIZED=18709876

run: 'net.sh eth0 start'

Unload module index

Unloaded link configuration context.

```

Although, it would seem to me, that my numvfs addition should appear in this output... shouldn't it?

At any rate, I edited my /etc/default/grub to remove

```
ixgbe.max_vfs=15
```

leaving the other iommu stuff intact, here is the kernel boot parameters:

```
GRUB_CMDLINE_LINUX_DEFAULT="splash=verbose,theme:emerge-world vga=795 net.ifnames=0 intel_iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1"
```

(and yes, I ran grub-mkconfig to put this change into effect).

Unforutnately   :Crying or Very sad:  this did not work I have NO virtual NIC interfaces after rebooting:

```
# cat /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0/device/sriov_numvfs

0
```

Summary - some progress, but now this feels like a path / syntax issue.

----------

## gordonp

I did find your suggested link helpful, @GDH-gentoo, but without success I began to wonder if my use of the "net" and "eth0" attributes may be too late in the boot-process, or even altogether wrong...

Google showed me this Debian link, with a variation on the rules:

http://enricorossi.org/blog/2016/intel_sr-iov_on_Debian_Stretch/

So, I re-configured my /etc/udev.d/rules like this:

```
KERNEL=="0000:01:00.0", SUBSYSTEM=="pci", DRIVER=="ixgbe", ATTR{vendor}=="0x8086", ATTR{device}=="0x10fb", ATTR{sriov_numvfs}="15"
```

And it works!

So, in summary:

```
# cat /etc/default/grub |grep -i GRUB_CMDLINE_LINUX_DEFAULT

GRUB_CMDLINE_LINUX_DEFAULT="splash=verbose,theme:emerge-world vga=795 net.ifnames=0 intel_iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1"
```

```
# cat /etc/udev/rules.d/eth0.rules

KERNEL=="0000:01:00.0", SUBSYSTEM=="pci", DRIVER=="ixgbe", ATTR{vendor}=="0x8086", ATTR{device}=="0x10fb", ATTR{sriov_numvfs}="15"
```

```
# dmesg -T |grep -i iov

[Thu Jan  2 14:45:23 2020] ixgbe 0000:01:00.0 eth0: SR-IOV enabled with 15 VFs

```

Last edited by gordonp on Thu Jan 02, 2020 10:54 pm; edited 1 time in total

----------

## GDH-gentoo

 *gordonp wrote:*   

> 
> 
> ```
> # udevadm test  /sys/class/net/eth0/
> 
> ...

 Ah, I guess that RHEL has some other rule that sets ID_NET_DRIVER, and Gentoo doesn't, at least with your installed packages, so this rule does not match. Try writing it like this:

```
ACTION=="add", SUBSYSTEM=="net", DRIVERS=="ixgbe", ATTR{device/sriov_numvfs}="15"
```

And retry the udevadm test command, to see what happens. If successful, there should be something like "ATTR [...] writing '15' /etc/udev/rules.d/eth0.rules:1" in the output.

----------

## gordonp

Yes... along the way, I found this command helpful:

 # udevadm info -a /sys/class/net/eth0/

Udevadm info starts with the device specified by the devpath and then

walks up the chain of parent devices. It prints for every device

found, all possible attributes in the udev rules key format.

A rule to match, can be composed by the attributes of the device

and the attributes from one single parent device.

... SNIP un-useful stuff ...

  looking at parent device '/devices/pci0000:00/0000:00:01.0/0000:01:00.0':

    KERNELS=="0000:01:00.0"

    SUBSYSTEMS=="pci"

    DRIVERS=="ixgbe"

    ATTRS{numa_node}=="-1"

    ATTRS{irq}=="27"

    ATTRS{local_cpulist}=="0-11"

    ATTRS{msi_bus}=="1"

    ATTRS{revision}=="0x01"

    ATTRS{ari_enabled}=="1"

    ATTRS{enable}=="1"

    ATTRS{driver_override}=="(null)"

ATTRS{sriov_numvfs}=="15"

    ATTRS{current_link_width}=="4"

    ATTRS{current_link_speed}=="5 GT/s"

    ATTRS{dma_mask_bits}=="64"

    ATTRS{class}=="0x020000"

    ATTRS{max_link_width}=="8"

    ATTRS{consistent_dma_mask_bits}=="64"

    ATTRS{local_cpus}=="0fff"

    ATTRS{sriov_drivers_autoprobe}=="1"

    ATTRS{sriov_offset}=="128"

    ATTRS{sriov_vf_device}=="10ed"

    ATTRS{subsystem_device}=="0x0006"

    ATTRS{sriov_totalvfs}=="63"

    ATTRS{device}=="0x10fb"

    ATTRS{subsystem_vendor}=="0x8086"

    ATTRS{vendor}=="0x8086"

    ATTRS{sriov_stride}=="2"

    ATTRS{broken_parity_status}=="0"

    ATTRS{max_link_speed}=="5 GT/s"

----------

