# permission denied on /dev/vfio/1 gpu passthrough kvm

## tse

Hello, I'm trying to achieve PCI-express GPU passthrough to a Window 10 VM, however, when trying to start the virtual machine (as root) with the hardware attached, this is what I get:

```

virsh # start win10-clone

error: Failed to start domain win10-clone

error: internal error: process exited while connecting to monitor: 2018-01-03T16:58:47.761436Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/4 (label charserial0)

2018-01-03T16:58:47.786124Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2: vfio error: 0000:01:00.0: failed to open /dev/vfio/1: Permission denied

```

The file however, exists:

```

# file /dev/vfio/1 

/dev/vfio/1: character special (250/0)

# ls -l /dev/vfio/1

crw------- 1 root root 250, 0 Jan  3 16:51 /dev/vfio/1

```

I followed the steps detailed in https://wiki.installgentoo.com/index.php/PCI_passthrough for host preparation, and in https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_the_guest_OS for setting up the guest.

Before running virsh, I:

```

/etc/init.d/libvirtd stop 

/etc/init.d/libvirt-guests stop

./vfio-bind 0000:01:00.0 0000:01:00.1

/etc/init.d/libvirtd start

/etc/init.d/libvirt-guests start

```

I've tried playing around with /etc/libvirt/qemu.conf, namely with the "group", "user", and "dynamic_ownership" settings - even though I'm running everything as root - and with "cgroup_device_acl" (by adding "/dev/vfio/1" to the list).

I cannot figure out what the problem in my setup is or what to try next. Is there anyone that can shine some light on this?

Thank you!

Following is the system information that I think is relevant for this problem:

lspci -vv | grep -i acscap

```

# lspci -vv | grep -i acscap

#

```

dmesg |  grep -e DMAR -e IOMMU

```

# dmesg |  grep -e DMAR -e IOMMU

[    0.000000] ACPI: DMAR 0x000000008A430B40 0000A8 (v01 INTEL  SKL      00000001 INTL 00000001)

[    0.000000] DMAR: IOMMU enabled

[    0.032444] DMAR: Host address width 39

[    0.032446] DMAR: DRHD base: 0x000000fed90000 flags: 0x0

[    0.032452] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e

[    0.032455] DMAR: DRHD base: 0x000000fed91000 flags: 0x1

[    0.032459] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da

[    0.032462] DMAR: RMRR base: 0x00000089f07000 end: 0x00000089f26fff

[    0.032476] DMAR: RMRR base: 0x0000008b800000 end: 0x0000008fffffff

[    0.032478] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1

[    0.032480] DMAR-IR: HPET id 0 under DRHD base 0xfed91000

[    0.033866] DMAR-IR: Enabled IRQ remapping in xapic mode

[    7.518966] DMAR: No ATSR found

[    7.519126] DMAR: dmar0: Using Queued invalidation

[    7.519131] DMAR: dmar1: Using Queued invalidation

[    7.519328] DMAR: Setting RMRR:

[    7.519348] DMAR: Setting identity map for device 0000:00:02.0 [0x8b800000 - 0x8fffffff]

[    7.519369] DMAR: Setting identity map for device 0000:00:14.0 [0x89f07000 - 0x89f26fff]

[    7.519375] DMAR: Prepare 0-16MiB unity mapping for LPC

[    7.519391] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

[    7.519412] DMAR: Intel(R) Virtualization Technology for Directed I/O

```

dmesg |grep vfio:

```

# dmesg |grep vfio

[    0.000000] Command line: BOOT_IMAGE=/kernel-genkernel-x86_64-4.9.72-gentoo root=/dev/mapper/vg0-root ro dolvm crypt_root=UUID=5016f5cf-65f7-4d8f-8c14-113c48778ad6 root=/dev/mapper/vg0-root iommu=on intel_iommu=on vfio_pci.ids=10de:1b81,10de:10f0 pcie_acs_override=downstream

[    0.000000] Kernel command line: BOOT_IMAGE=/kernel-genkernel-x86_64-4.9.72-gentoo root=/dev/mapper/vg0-root ro dolvm crypt_root=UUID=5016f5cf-65f7-4d8f-8c14-113c48778ad6 root=/dev/mapper/vg0-root iommu=on intel_iommu=on vfio_pci.ids=10de:1b81,10de:10f0 pcie_acs_override=downstream

[    7.627739] vfio_pci: add [10de:1b81[ffff:ffff]] class 0x000000/00000000

[    7.639685] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000

```

lspci -nnk | grep -A3 -i nvidia

```

# lspci -nnk | grep -A3 -i nvidia

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)

   Subsystem: Micro-Star International Co., Ltd. [MSI] GP104 [GeForce GTX 1070] [1462:3302]

   Kernel driver in use: vfio-pci

   Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)

   Subsystem: Micro-Star International Co., Ltd. [MSI] GP104 High Definition Audio Controller [1462:3302]

   Kernel driver in use: vfio-pci

   Kernel modules: snd_hda_intel

```

./iommu_groups.sh

```

# ./iommu_groups.sh

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:191f] (rev 07)

IOMMU Group 10 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8] (rev 31)

IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)

IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)

IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)

IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06)

IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)

IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)

IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)

IOMMU Group 6 00:1b.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #17 [8086:a167] (rev f1)

IOMMU Group 6 00:1b.2 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #19 [8086:a169] (rev f1)

IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)

IOMMU Group 7 00:1c.1 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #2 [8086:a111] (rev f1)

IOMMU Group 7 00:1c.2 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #3 [8086:a112] (rev f1)

IOMMU Group 7 00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)

IOMMU Group 7 07:00.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]

IOMMU Group 7 08:00.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]

IOMMU Group 7 08:01.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]

IOMMU Group 7 08:02.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]

IOMMU Group 7 08:04.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]

IOMMU Group 7 0b:00.0 USB controller [0c03]: Intel Corporation DSL6540 USB 3.1 Controller [Alpine Ridge] [8086:15b6]

IOMMU Group 8 00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)

IOMMU Group 8 00:1d.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #13 [8086:a11c] (rev f1)

IOMMU Group 9 00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a145] (rev 31)

IOMMU Group 9 00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)

IOMMU Group 9 00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)

IOMMU Group 9 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)

```

cat win10-clone.xml

```

# cat win10-clone.xml

<!--

WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE

OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:

  virsh edit win10-clone

or other application using the libvirt API.

-->

<domain type='kvm'>

  <name>win10-clone</name>

  <uuid>a673c9c8-a1f5-4c3d-97a7-16096bcf1b01</uuid>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>4194304</currentMemory>

  <vcpu placement='static'>2</vcpu>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>

    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>

    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

    <hyperv>

      <relaxed state='on'/>

      <vapic state='on'/>

      <spinlocks state='on' retries='8191'/>

    </hyperv>

    <vmport state='off'/>

  </features>

  <cpu mode='host-model' check='partial'>

    <model fallback='allow'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

    <timer name='hypervclock' present='yes'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <pm>

    <suspend-to-mem enabled='no'/>

    <suspend-to-disk enabled='no'/>

  </pm>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/home/tse/kvm_vm/test1-clone.qcow2'/>

      <target dev='sda' bus='scsi'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <target dev='hdb' bus='ide'/>

      <readonly/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <target dev='hdc' bus='ide'/>

      <readonly/>

      <address type='drive' controller='0' bus='1' target='0' unit='0'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci2'>

      <master startport='2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci3'>

      <master startport='4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>

    </controller>

    <controller type='scsi' index='0' model='virtio-scsi'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </controller>

    <interface type='direct'>

      <mac address='52:54:00:45:6f:82'/>

      <source dev='enp0s31f6' mode='bridge'/>

      <model type='rtl8139'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <sound model='ac97'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </sound>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <source>

        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <source>

        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>

    </hostdev>

    <redirdev bus='usb' type='spicevmc'>

      <address type='usb' bus='0' port='2'/>

    </redirdev>

    <redirdev bus='usb' type='spicevmc'>

      <address type='usb' bus='0' port='3'/>

    </redirdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

    </memballoon>

  </devices>

</domain>

```

cat iommu_groups.sh 

```

# cat iommu_groups.sh 

#!/bin/bash

shopt -s nullglob

for d in /sys/kernel/iommu_groups/*/devices/*; do 

    n=${d#*/iommu_groups/*}; n=${n%%/*}

    printf 'IOMMU Group %s ' "$n"

    lspci -nns "${d##*/}"

done;

```

cat vfio-bind 

```

# cat vfio-bind 

#!/bin/bash 

modprobe vfio-pci

for dev in "$@"; do

        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)

        device=$(cat /sys/bus/pci/devices/$dev/device)

        if [ -e /sys/bus/pci/devices/$dev/driver ]; then

                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind

        fi

        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

   echo "done"

done

```

emerge --pretend --verbose qemu

```

[ebuild   R    ] app-emulation/qemu-2.10.1-r1::gentoo  USE="aio alsa bzip2 caps curl fdt filecaps jpeg ncurses nls opengl pin-upstream-blobs png python seccomp spice usb usbredir vhost-net vnc xattr -accessibility -bluetooth -debug (-glusterfs) -gnutls -gtk -gtk2 -infiniband -iscsi -lzo -nfs -numa -pulseaudio -rbd -sasl -sdl -sdl2 (-selinux) -smartcard -snappy -ssh -static -static-user -systemtap -tci {-test} -vde -virgl -virtfs -vte -xen -xfs" LINGUAS="-bg -de_DE -fr_FR -hu -it -tr -zh_CN" PYTHON_TARGETS="python2_7" QEMU_SOFTMMU_TARGETS="x86_64 -aarch64 -alpha -arm -cris -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -moxie -nios2 -or1k -ppc -ppc64 -ppcemb -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="-aarch64 -alpha -arm -armeb -cris -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -tilegx -x86_64"

```

----------

## blopsalot

i would try starting with a simpler qemu command, try to turn off kvm. make sure vfio grabbing device first. im not familar with libvirt tho.

----------

