# [SOLVED] Setting up IOMMU and QEMU for VGA Passthrough

## NilPointer

I'm trying to achieve the VGA Passthrough. I've got following hardware:

Gigabyte GA-990X-GAMING SLI with IOMMU controller.

AMD FX-8350 CPU with IOMMU support.

Two AMD Radeon RX480 GPUs.

I want to give the spare GPU to the VM with PCI passthrough. I've enabled IOMMU in BIOS, compiled the kernel with relevant modules and cmdline options.

This is my customized cmdline for kernel:

```
GRUB_CMDLINE_LINUX="max_loop=64 iommu=pt iommu=1 amd_iommu=fullflush"
```

And these lines appear in dmesg:

```
[    1.176991] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

[    1.177107] AMD-Vi: Interrupt remapping enabled

[    1.177302] AMD-Vi: IO/TLB flush on unmap enabled

...

[    1.089694] iommu: Adding device 0000:00:00.0 to group 0

[    1.089818] iommu: Using direct mapping for device 0000:00:00.0

[    1.090242] iommu: Adding device 0000:00:02.0 to group 1

[    1.090398] iommu: Using direct mapping for device 0000:00:02.0

[    1.090820] iommu: Adding device 0000:00:03.0 to group 2

[    1.090941] iommu: Using direct mapping for device 0000:00:03.0

[    1.091371] iommu: Adding device 0000:00:09.0 to group 3

[    1.091519] iommu: Using direct mapping for device 0000:00:09.0

[    1.091941] iommu: Adding device 0000:00:11.0 to group 4

[    1.092063] iommu: Using direct mapping for device 0000:00:11.0

[    1.092507] iommu: Adding device 0000:00:12.0 to group 5

[    1.092630] iommu: Using direct mapping for device 0000:00:12.0

[    1.092784] iommu: Adding device 0000:00:12.2 to group 5

[    1.093215] iommu: Adding device 0000:00:13.0 to group 6

[    1.093343] iommu: Using direct mapping for device 0000:00:13.0

[    1.093471] iommu: Adding device 0000:00:13.2 to group 6

[    1.093895] iommu: Adding device 0000:00:14.0 to group 7

[    1.094043] iommu: Using direct mapping for device 0000:00:14.0

[    1.094471] iommu: Adding device 0000:00:14.2 to group 8

[    1.094592] iommu: Using direct mapping for device 0000:00:14.2

[    1.095014] iommu: Adding device 0000:00:14.3 to group 9

[    1.095163] iommu: Using direct mapping for device 0000:00:14.3

[    1.098073] iommu: Adding device 0000:00:14.4 to group 10

[    1.098194] iommu: Using direct mapping for device 0000:00:14.4

[    1.098629] iommu: Adding device 0000:00:14.5 to group 11

[    1.098750] iommu: Using direct mapping for device 0000:00:14.5

[    1.099207] iommu: Adding device 0000:00:15.0 to group 12

[    1.099333] iommu: Using direct mapping for device 0000:00:15.0

[    1.099463] iommu: Adding device 0000:00:15.3 to group 12

[    1.099894] iommu: Adding device 0000:00:16.0 to group 13

[    1.100039] iommu: Using direct mapping for device 0000:00:16.0

[    1.100167] iommu: Adding device 0000:00:16.2 to group 13

[    1.100627] iommu: Adding device 0000:01:00.0 to group 14

[    1.100750] iommu: Using direct mapping for device 0000:01:00.0

[    1.100891] iommu: Adding device 0000:01:00.1 to group 14

[    1.101359] iommu: Adding device 0000:02:00.0 to group 15

[    1.101485] iommu: Using direct mapping for device 0000:02:00.0

[    1.101631] iommu: Adding device 0000:02:00.1 to group 15

[    1.102056] iommu: Adding device 0000:03:00.0 to group 16

[    1.102176] iommu: Using direct mapping for device 0000:03:00.0

[    1.102297] iommu: Adding device 0000:04:06.0 to group 10

[    1.102452] iommu: Adding device 0000:05:00.0 to group 12

[    1.102578] iommu: Adding device 0000:06:00.0 to group 12

```

The GPUs are there:

```
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 67df (rev c7)

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0

02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 67df (rev c7)

02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0
```

They're both being bound to amdgpu driver on boot and that's unwanted. How can I bind 02.00.0 to vfio-pci driver? Apparently, it's easier to bind devices with different vendor/device IDs. But in my case, there's two GPUs of same kind.

I've tried unbinding device with following script and rebinding it to vfio-pci driver manually:

```
#!/bin/sh

modprobe -i vfio-pci

echo 0000:02:00.0 > /sys/bus/pci/drivers/amdgpu/unbind

echo 0000:02:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
```

Running these commands cause nothing unusual in the first 5 seconds, then everything starts to stutter (mouse, probably, moves 1 frame per second) and few seconds later, machine completely deadlocks, not responding to mouse, keyboard or SSH. How could I force vfio-pci driver for second GPU? Am I doing something wrong?Last edited by NilPointer on Mon Aug 29, 2016 8:28 pm; edited 1 time in total

----------

## szatox

I'd try blacklisting radeon so it doesn't load at boot.

Bind one GPU to virtio (pci_stub module? I'm not quite sure which one should bind it) and then load radeon so it will claim the other, unused device.

----------

## NilPointer

Looks like I've made it. Thanks for advice. I've blacklisted amdgpu driver and removed amdgpu from /etc/conf.d/modules. Then, I've used the following script, which I've found there

http://vfio.blogspot.ru/2015/05/vfio-gpu-how-to-series-part-3-host.html:

```
#!/bin/sh

for i in $(find /sys/devices/pci* -name boot_vga); do

        if [ $(cat $i) -eq 0 ]; then

                GPU=$(dirname $i)

                AUDIO=$(echo $GPU | sed -e "s/0$/1/")

                echo "vfio-pci" > $GPU/driver_override

                if [ -d $AUDIO ]; then

                        echo "vfio-pci" > $AUDIO/driver_override

                fi

        fi

done

modprobe -i vfio-pci
```

That adds driver override for non-boot VGA adapter (which is 0000:02:00.0 in my case, and to it's companion HDMI audio) and loads vfio-pci, which binds to those. Then I can modprobe amdgpu which will bind to boot adapter and load XOrg. I think I'll add that script to my boot sequence (with "modprobe amdgpu" appended).

Finally, qemu starts (although only as root) and second GPU actually outputs signal! Woo-hoo! Guess that means GPU passthrough works!

----------

