# PCI passthrough - VFIO and QEMU.

## Kresp

I'm trying to pass GTX 960 GPU to WIn7 VM.

Currently I'm running single-GPU setup to simplify, since my motherboard lumps both GPU slots into one IOMMU group.

I'm trying to follow this:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_IOMMU

https://wiki.installgentoo.com/index.php/PCI_passthrough

This is my IOMMU 2 group with GPU:

```

IOMMU group 2

   00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

   00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]

   25:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)

   25:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)

```

I blacklisted nouveau and snd_hda_intel and added following to /etc/modprobe.d/vfio.conf:

```

options vfio-pci ids=10de:1401,10de:0fba

```

After reboot, PC does not init KMS mode anymore but there's still boot log in fallback VGA mode.

Now I ssh into the computer from another machine. Turns out, gpu device does not have any modules loaded. So I do modprobe vfio-pci, which loads correctly for both video and audio devices.

At this point monitor freezes (login prompt stops blinking).

Now, I try to launch qemu with following arguments (VM itself is ok, it normally runs fine with qemu VGA adapter without the last three lines):

```

qemu-system-x86_64 \

        -drive file=~/vm/win7.qcow2,if=virtio \

        \

        -enable-kvm \

        -cpu host,kvm=off \

        -smp cpus=8,maxcpus=8,cores=8,threads=1,sockets=1 \

        \

        -m 16384 \

        \

        -device vfio-pci,host=25:00.0,x-vga=on \

        -device vfio-pci,host=25:00.1 \

        -vga none

```

It complains about gtk (X is not running), so I re-emerge with -gtk -vte.

By this point VM launches and judging by htop stats, boots fine. However, display still shows frozen boot log.

Now, if I go back and revert everything, then I can not use this VM with qemu VGA adapter until I re-emerge package with +gtk again.

I tried adding video=efifb:off to GRUB kernel line. Now boot log stops after the following (but it changes nothing else):

```

Loading Linux 4.10.6-gentoo ...

Loading initial ramdisk ...

```

----------

## Myu

Hello !

I wish I could be able to help (I also run a Gentoo/VFIO setup, here with Intel HD + GTX750 Ti) but I don't really have any knowledge about a single GPU passthrough, it indeed add quite a bit of complexity to the debugging.

That being said, a few things :

- Do you boot your VM with EFI ? Afaik you'll have better results with that instead of BIOS (the general idea is that pci-stub+BIOS = old method, UEFI + VFIO = new)

- You should make sure your card loads the vfio_pci module at boot time, perhaps the modules service is not started at boot time ? Check rc-config list

-  Also regarding the modules service : /etc/conf.d/modules, this file contains this on my end (among other commented things) : 

```

modules="vfio vfio-pci vfio_iommu_type1 vfio_virqfd"
```

- A very good source of information and help is available at https://reddit.com/r/vfio "lots" (relatively speaking  :Mr. Green:  ) of people with working setups.

Let me know if I can help you further, I've some docs but aimed at and Intel/Nvidia setup.

Cheers,

- Myu

----------

## Kresp

```
$ rc-config list | grep -i modules

  modules                   boot

  modules-load

```

I currently use Seabios. Where do I get OVMF binaries? It does not seem to be in the portage overlay.

----------

## Myu

I've grabbed mine from the builds available there : https://www.kraxel.org/repos/jenkins/edk2/, specifically the edk2.git-ovmf-x64-0-*.noarch.rpm

----------

## Kresp

So, I grabbed UEFI Win7 installation, and reinstalled it with OVMF efi.

Now, when I start VM, monitor instead of keeping frozen bootlog, blackens out and goes to standy mode (as if PC is powered down).

Qemu reports that it failed to mmap to device ID.

So I grep'd over /proc/iomem and found out that efifb is holding some of device memory.

After recompiling without FB_EFI, this warning is gone, but monitor still does not display a thing.

----------

## archenroot

It should be worth to write an article on Gentoo wiki about this. I tried some time ago on laptop without luck.

----------

## Quazil

Windows 7 is particularly difficult to install with GPU pass-through.

I have not gotten Windows 7 to work. I ended up going to Windows 10.

> After reboot, PC does not init KMS mode anymore but there's still boot log in fallback VGA mode. 

Yeah you can't do that.

You'd have to boot the host OS in headless mode if you only have 1 video card.  

The other video device does not have to be a GPU but whatever it is, you have to boot using it.

----------

## dylanmc

Hi to all, I've bought a new pc and I want to get VGA passtrought to work.

I've two gpus, and I want to buy a new monitor with two hdmi ports, in order to bypass the two monitor space problem.

I'm read some guides, but for other distros.

```
luca@snoopx ~ $ dmesg |grep IOMMU

[    0.288498] AMD-Vi: IOMMU performance counters supported

[    0.289165] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

[    0.289894] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

[    0.302138] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

luca@snoopx ~ $ lsmod 

Module                  Size  Used by

vboxdrv               372736  0

nvidia_modeset        843776  9

vfio_pci               40960  0

vfio_virqfd            16384  1 vfio_pci

vfio_iommu_type1       24576  0

vfio                   28672  2 vfio_iommu_type1,vfio_pci

tuner_xc2028           32768  1

zl10353                16384  1

em28xx_dvb             36864  0

hid_generic            16384  0

kvm_amd              2166784  0

kvm                   368640  1 kvm_amd

irqbypass              16384  2 kvm,vfio_pci

em28xx                 90112  1 em28xx_dvb

tveeprom               24576  1 em28xx

usbhid                 32768  0

hid                   110592  3 hid_generic,usbhid

nvidia              13107200  1159 nvidia_modeset

nvidia_drm             16384  0

```

I've read that if all works fine, virt-pci will be in use by my gpu passtrought card, but I didin't see it.

```
lspci -nnv 

3:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1) (prog-if 00 [VGA controller])

        Subsystem: Gigabyte Technology Co., Ltd GM200 [GeForce GTX 980 Ti] [1458:36b6]

        Flags: bus master, fast devsel, latency 0, IRQ 45

        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]

        Memory at d0000000 (64-bit, prefetchable) [size=256M]

        Memory at e0000000 (64-bit, prefetchable) [size=32M]

        I/O ports at f000 [size=128]

        [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]

        Capabilities: [60] Power Management version 3

        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [78] Express Legacy Endpoint, MSI 00

        Kernel driver in use: nvidia

        Kernel modules: nvidiafb, nvidia_drm, nvidia

```

Virt manager says that all ok

```

noopx luca # virt-host-validate 

  QEMU: Checking for hardware virtualization                                 : PASS

  QEMU: Checking if device /dev/kvm exists                                   : PASS

  QEMU: Checking if device /dev/kvm is accessible                            : PASS

  QEMU: Checking if device /dev/vhost-net exists                             : PASS

  QEMU: Checking if device /dev/net/tun exists                               : PASS

  QEMU: Checking for cgroup 'memory' controller support                      : PASS

  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS

  QEMU: Checking for cgroup 'cpu' controller support                         : PASS

  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS

  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS

  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS

  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS

  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS

  QEMU: Checking for cgroup 'devices' controller support                     : PASS

  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS

  QEMU: Checking for cgroup 'blkio' controller support                       : PASS

  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS

  QEMU: Checking for device assignment IOMMU support                         : PASS

  QEMU: Checking if IOMMU is enabled by kernel                               : PASS

snoopx luca # 

```

any suggestion?

----------

## Myu

Hello dylanmc,

```

Kernel driver in use: nvidia
```

It seems that your 980 is currently using the nvidia driver while it should use the vfio-pci one

Did you added the various ID's to /etc/motdporbe.d/vfio.conf ? here's mine for reference :

```
cat /etc/modprobe.d/vfio.conf 

options vfio-pci ids=10de:1380,10de:0fbc,8086:1e20
```

Here's the longer version on how to isolate the GPU :

Get your vendor-id :

    lspci | grep -i vga

> note down the first number, it is the slot number i.e 01:00.0

    lspci -nns 01:00.0

> Note down the last number between "[]", this is the vendor-id, i.e 10de:1380

    lspci -nnk -d vendor-id

Edit `/etc/modprobe.d/vfio.conf` with the vendor-id from your GPU you want to isolate gathered from the previous command, in this __example__, the vendor-id is __10de:13c2__ for the GPU and __10de:0fbb__ for the audio

    options vfio-pci ids=10de:13c2,10de:0fbb

Add these modules to `/etc/conf.d/modules` (Gentoo/OpenRC specific)

    modules="vfio vfio-pci vfio_iommu_type1 vfio_virqfd"

module loading at boot is enabled by

        rc-update add modules boot

Hope this helps  :Smile: 

----------

## dylanmc

Thanks for the reply, but I sill don't understand where do you get 

```
8086:1e20
```

 in your /etc/conf.d/vfio.conf

That's mine

```
luca@snoopx ~ $ cat /etc/modprobe.d/vfio.conf 

options vfio-pci ids=10de:17c8,10de:0fb0
```

where 17c8 is for the video, 0fb0 for the audio.

Infact, the driver in use for my 980ti is still the nvidia one kernel driver.

----------

## dylanmc

my lsmod

```
luca@snoopx ~ $ lsmod 

Module                  Size  Used by

nvidia_modeset        843776  8

vfio_pci               40960  0

vfio_virqfd            16384  1 vfio_pci

vfio_iommu_type1       24576  0

vfio                   28672  2 vfio_iommu_type1,vfio_pci

vboxpci                24576  0

vboxnetadp             28672  0

vboxnetflt             28672  0

vboxdrv               372736  3 vboxnetadp,vboxnetflt,vboxpci

tuner_xc2028           32768  1

zl10353                16384  1

em28xx_dvb             36864  0

hid_generic            16384  0

kvm_amd              2166784  0

kvm                   368640  1 kvm_amd

irqbypass              16384  2 kvm,vfio_pci

em28xx                 90112  1 em28xx_dvb

tveeprom               24576  1 em28xx

usbhid                 32768  0

hid                   110592  3 hid_generic,usbhid

nvidia              13107200  878 nvidia_modeset

nvidia_drm             16384  0

```

----------

## Myu

Ah, nevermind the 8086:1e20, it's the MB audio chipset that I'm passing through to the VM as well, nothing to do with the GPU  :Smile: 

So if you reboot your Gentoo, the nvidia driver get picked up even with vfio properly set to load at boot time ? (i.e the part of my previous post with /etc/conf.d/modules)

What does dmesg | grep -i vfio gives ? A proper isolation should look like this :

```

[    1.886071] vfio_pci: add [10de:1380[ffff:ffff]] class 0x000000/00000000

[    1.898075] vfio_pci: add [10de:0fbc[ffff:ffff]] class 0x000000/00000000

[    1.911066] vfio_pci: add [8086:1e20[ffff:ffff]] class 0x000000/00000000
```

Are you sure you're properly booting to the second card when that happens ?

----------

## dylanmc

```
snoopx luca #  dmesg | grep -i vfio 

[    0.000000] Command line: BOOT_IMAGE=4.13.13 ro root=/dev/sda3 rootfstype=ext4 iommu=1 amd_iommu=on rd.driver.pre=vfio-pci

[    0.000000] Kernel command line: BOOT_IMAGE=4.13.13 ro root=/dev/sda3 rootfstype=ext4 iommu=1 amd_iommu=on rd.driver.pre=vfio-pci

[    3.861433] VFIO - User Level meta-driver version: 0.3

[    3.870849] vfio_pci: add [10de:17c8[ffff:ffff]] class 0x000000/00000000

[    3.870851] vfio_pci: add [10de:0fb0[ffff:ffff]] class 0x000000/00000000
```

It seems fine.

 *Quote:*   

> Are you sure you're properly booting to the second card when that happens ?

 

Actually I'm still booting with the primary device card, the GT980ti.

If I have understood well, I've to plug my monitor to the 710 nvidia card, and I will get my GPU 980ti isolated? (and with the vfio-pci module ,loaded instead of the nvidia one?)

----------

## Myu

 *Quote:*   

> Actually I'm still booting with the primary device card, the GT980ti. 

 

It shouldn't be the case if you wish to isolate that card, you'll have to set your BIOS/UEFI to boot to the 710 and hook it to a monitor, yes.

----------

## dylanmc

I'm ok with it.

But I've tried to isolate de 710b, and I have booted as usual from the 980ti.

I still get that the two cards are managed by the nvidia driver.

Maybe the nvidiafb can create issues?

I've to create an initramfs?

PS: i'm still using lilo, but I don't think is the problem.

----------

## Myu

Good question(s), I'm not super knowledgable about a setup with two nvidia cards but yeah, if you can scratch nvidiafb from the equation, that might help.

Also, can you clarify what card is set to primary on your BIOS/UEFI ?

I'm not using an initramfs personally.

----------

## dylanmc

```
Also, can you clarify what card is set to primary on your BIOS/UEFI ? 
```

In my bios I don't find any type of settings about which card is set to primary or else.

The nvidia-fb was already blacklisted,because he was blocking my system to recognize the two nvidia cards (with nvidafb I see only the 980ti)

mybee I have to load vfio-pci before the nvidia one?

Bit I don't know to do it.. :/

----------

## Myu

 *Quote:*   

> In my bios I don't find any type of settings about which card is set to primary or else. 

 

Ah, then it's likely determined by where you plug it (the PCIe port), there has to be a kind of "master" one to figure out.

As far as I know, adding the proper ID's to /etc/modprobe.d/vfio.conf instruct the vfio_pci kernel module to be loaded on the GPU instead of nvidia.

Is the module "service" properly set to load at boot time ? If you're on OpenRC, you can check with :

```
# rc-config list | grep module

  modules                   boot

  modules-load

```

I would recheck this step (again OpenRC specific, adjust if you're on systemd) :

Add these modules to /etc/conf.d/modules

```
    modules="vfio vfio-pci vfio_iommu_type1 vfio_virqfd"
```

module loading at boot is enabled by

```
        rc-update add modules boot
```

Hope this helps !

----------

## dylanmc

```
snoopx luca # rc-config list | grep module 

  modules                   boot

  modules-load

```

Modules loading is enabled, I  load also the modules for virtualbox.

I don't think that those modules conflict with vfio-pci.

 *Quote:*   

> As far as I know, adding the proper ID's to /etc/modprobe.d/vfio.conf instruct the vfio_pci kernel module to be loaded on the GPU instead of nvidia. 

 

I'm sure I'm passing the corect IDS to the vfio-pici mdules, but all the two cards are with the nvidia module  :Sad: 

----------

## dylanmc

Another thing

If I execute

```
noopx luca # modinfo vfio-pci

filename:       /lib/modules/4.13.13-gentoo/kernel/drivers/vfio/pci/vfio-pci.ko

description:    VFIO PCI - User Level meta-driver

author:         Alex Williamson <alex.williamson@redhat.com>

license:        GPL v2

version:        0.2

srcversion:     DB8F55EC2187EC83F7E71EA

depends:        vfio,irqbypass,vfio_virqfd

intree:         Y

name:           vfio_pci

vermagic:       4.13.13-gentoo SMP mod_unload 

parm:           ids:Initial PCI IDs to add to the vfio driver, format is "[b]vendor:device[:subvendor[:subdevice[:class[:class_mask[/b]]]]]" and multiple comma separated entries can be specified (string)

parm:           nointxmask:Disable support for PCI 2.3 style INTx masking.  If this resolves problems for specific devices, report lspci -vvvxxx to linux-pci@vger.kernel.org so the device can be fixed automatically via the broken_intx_masking flag. (bool)

parm:           disable_vga:Disable VGA resource access through vfio-pci (bool)

parm:           disable_idle_d3:Disable using the PCI D3 low power state for idle, unused devices (bool)

```

maybe I can try adding subvendor/subdevice ?

----------

## dylanmc

Finally I got this working.

I've do this

```
options vfio-pci ids=10de:128b,10de:0e0f

softdep nvdia pre: vfio-pci

softdep nvidia_modeset pre: vfio-pci

softdep nvidia_drm pre: vfio-pci

```

in the file /etc/conf.d/vfio.conf

and this

```

mage=/boot/vmlinuz-4.13.13-gentoo

  label=4.13.13

  read-only

  append="root=/dev/sda3 rootfstype=ext4 iommu=1 amd_iommu=on rd.driver.pre=[b]vfio-pci.ids=10de:128b,10de:0e0f[/b]"

```

in /etc/lilo.conf

```
22:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1) (prog-if 00 [VGA controller])

        Subsystem: ASUSTeK Computer Inc. GK208 [GeForce GT 710B] [1043:8576]

        Flags: fast devsel, IRQ 11

        Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=16M]

        Memory at e8000000 (64-bit, prefetchable) [disabled] [size=128M]

        Memory at f0000000 (64-bit, prefetchable) [disabled] [size=32M]

        I/O ports at d000 [disabled] [size=128]

        Expansion ROM at f5000000 [disabled] [size=512K]

        Capabilities: [60] Power Management version 3

        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+

        Capabilities: [78] Express Legacy Endpoint, MSI 00

        Kernel driver in use:[b] vfio-pci[/b]

        Kernel modules: nvidiafb, nvidia_drm, nvidia

```

Thanks for the help! now I have to set qemu, but I have to buy a second monitor first.

For emulation,  do you use virt-manager or a line command qemu?

----------

## Myu

Cool, I'm glad you managed to get it working and took the time to document it here  :Smile: 

I personally use virt-manager, it provide a nice abstraction to the qemu cli, making it easy to select your 980 once it's binded to the vfio_pci driver but also other components (HD Audio, ...)

Many ofther vfio users are using virt-manager so there's quite some documentation available (see reddit.com/r/vfio)

There's a few quirks regarding setting up Windows with a nvidia card since nvidia basically doesn't want you to do that and throws an infamous error 43, you have to hide the fact you're using an hypervisor with this (if you're using virt-manger)

```
# virsh edit vm_name
```

Add inside the <features> </features> tag :

```

    <kvm>

      <hidden state='on'/>

    </kvm>

```

Happy setup !

----------

## dylanmc

The next step is create a virbr0, virt-manager is asking me for setting up a new vm.

I've to isolate also the network interface?

----------

## Myu

It's up to you for the networking, you can setup a bridge like virbr0 or if you have the hardware, you can passthrough a wifi dongle, an ethernet adapter, ...

----------

## dylanmc

Now I'm tryng to start a WM, with virt manager but I get some errors  :Sad: 

I'f I select "customize installation" I get this error

```
Errore nell'avvio dell'installazione: 'nome-mv'

Traceback (most recent call last):

  File "/usr/share/virt-manager/virtManager/create.py", line 2199, in _finish_clicked

    self._show_customize_dialog(guest)

  File "/usr/share/virt-manager/virtManager/create.py", line 2230, in _show_customize_dialog

    self._customize_window = vmmDetails(virtinst_guest, self.topwin)

  File "/usr/share/virt-manager/virtManager/details.py", line 363, in __init__

    self.console = vmmConsolePages(self.vm, self.builder, self.topwin)

  File "/usr/share/virt-manager/virtManager/console.py", line 128, in __init__

    self._change_title()

  File "/usr/share/virt-manager/virtManager/console.py", line 291, in _change_title

    "connection-name": self.vm.conn.get_pretty_desc(),

KeyError: 'nome-mv'
```

I'm not sure, but  I think it's a python error  :Sad: 

[/code]

----------

## bunder

I found virt-manager to be picky in the config options it allows.  If you have to do something it doesn't like, you wind up using virsh edit, and using virt-manager to start/stop vms only.

----------

## dylanmc

whitout a virt-manager I'm pretty sure I'm not able to run my qemu WM with the gpu passtrought, I think It's too hard for me  :Sad: 

----------

## Hu

For what it's worth, you should not need to use virt-manager at all for GPU passthrough.  With the right options, Qemu can do that even when called directly.  However, finding those options without virt-manager may be more trouble than you want to undertake.

----------

## dylanmc

With virt-manager I get those errors

```
rrore nell'avvio dell'installazione: 'nome-mv'

Traceback (most recent call last):

  File "/usr/share/virt-manager/virtManager/create.py", line 2199, in _finish_clicked

    self._show_customize_dialog(guest)

  File "/usr/share/virt-manager/virtManager/create.py", line 2230, in _show_customize_dialog

    self._customize_window = vmmDetails(virtinst_guest, self.topwin)

  File "/usr/share/virt-manager/virtManager/details.py", line 363, in __init__

    self.console = vmmConsolePages(self.vm, self.builder, self.topwin)

  File "/usr/share/virt-manager/virtManager/console.py", line 128, in __init__

    self._change_title()

  File "/usr/share/virt-manager/virtManager/console.py", line 291, in _change_title

    "connection-name": self.vm.conn.get_pretty_desc(),

KeyError: 'nome-mv'
```

I've search in google, but no luck.

I don't understand if is a virt manager error, or a python issue.

----------

## dylanmc

Solved that issue upgrating to version 1.4.3 of virt-manager  :Smile: 

Now another problem,

```
libvirtError: internal error: process exited while connecting to monitor: 2017-12-17T19:32:39.826145Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)

2017-12-17T19:32:39.843865Z qemu-system-x86_64: -device vfio-pci,host=22:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio error: 0000:22:00.0: group 0 is not viable

Please ensure all devices within the iommu_group are bound to their vfio bus driver.
```

What does it means group 0 is not viable?

----------

## Myu

It means all the devices you're trying to passthrough from group 0 are not bound to the vfio_pci driver/not suitable for passthrough

How are your IOMMU groups looking ?

```

#!/bin/bash

shopt -s nullglob

for d in /sys/kernel/iommu_groups/*/devices/*; do

    n=${d#*/iommu_groups/*}; n=${n%%/*}

    printf 'IOMMU Group %s ' "$n"

    lspci -nns "${d##*/}"

done;

```

----------

## dylanmc

```
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]

IOMMU Group 0 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bb] (rev 02)

IOMMU Group 0 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b7] (rev 02)

IOMMU Group 0 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)

IOMMU Group 0 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)

IOMMU Group 0 04:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)

IOMMU Group 0 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)

IOMMU Group 0 1f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 03)

IOMMU Group 0 21:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

IOMMU Group 0 22:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)

IOMMU Group 0 22:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]

IOMMU Group 2 23:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)

IOMMU Group 2 23:00.1 Audio device [0403]: NVIDIA Corporation GM200 High Definition Audio [10de:0fb0] (rev a1)

IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]

IOMMU Group 4 24:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]

IOMMU Group 4 24:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]

IOMMU Group 4 24:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]

IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]

IOMMU Group 5 25:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]

IOMMU Group 5 25:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

IOMMU Group 5 25:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]

IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)

IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)

IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]

IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]

IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]

IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]

IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]

IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]

IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]

IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]

```

----------

## Myu

Are you on the latest BIOS (AGESA) ? I've read that v1.0.0.7 improved groupings a lot.

As it is, these groups are not great for passthrough, too much stuff in it.

Side question : Which item from group 0 are you trying to passthrough ?

----------

## dylanmc

```
IOMMU Group 0 22:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1) 

IOMMU Group 0 22:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1) 
```

I have to verify if it's the lastest bios, but it's probably not the last...

----------

## dylanmc

 *Myu wrote:*   

> Are you on the latest BIOS (AGESA) ? I've read that v1.0.0.7 improved groupings a lot.
> 
> As it is, these groups are not great for passthrough, too much stuff in it.
> 
> Side question : Which item from group 0 are you trying to passthrough ?

 

maybe can I have much luck isolating the 980ti, in group 2 ?

Because it's the real card I want to isolate, now I was only making a try.

The bios is the lastest, 1.0.0.6b ... nothing about 1.0.0.7 for now..

----------

## Myu

I'm not sure you'll have better luck with the 980ti group 2, still too much stuff in it but you can try to be certain.

For instance, here's my working 750 Ti groupings, there's only the VGA-compatible device and the GPU Audio chip :

```
IOMMU Group 12 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)

IOMMU Group 12 02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
```

The reference for AGESA 1.0.0.7 improving IOMMU groups is here : https://www.reddit.com/r/VFIO/comments/7dpsva/new_iommu_groups_on_ryzen_agesa_1007/

----------

## dylanmc

I will try, actually I have nothing else to do  :Sad: 

I hope in a AGESA 1.0.0.7 soon ...  now I'm thinking about that I have to buy an intel processor  :Sad: 

----------

## Myu

I hope that you'll get it soon, may I ask you which motherboard do you have ?

Otherwise, there's already some motherboards with AGESA v1.0.0.7 but I can imagine it's a hassle.

----------

## dylanmc

Actually I have and MSI Tomahawk B350, buyed 2 week ago..  I have to wait, no other choices  :Razz: 

----------

## dylanmc

Another question.

I see that 

```
OMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]

IOMMU Group 2 23:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
```

Seems to be almost usable.. but the is a way to boot up on the secondary vga?

I've tryed with this parm to kernel, with no luck 

```

video=vesafb:off,efifb:off
```

----------

## dylanmc

I've got my initial MB back, and I decide to remount it.

It's a ASUS B350 prime plus, updated to AGESA 1071 but ... 

```
snoopx /home/luca # ./iommu                                                                                                                                           

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]                                                                       

IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]                                                                        

IOMMU Group 0 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bb] (rev 02)                                                           

IOMMU Group 0 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b7] (rev 02)                                                          

IOMMU Group 0 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)                                                               

IOMMU Group 0 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)

IOMMU Group 0 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)

IOMMU Group 0 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)

IOMMU Group 0 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

IOMMU Group 0 04:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)

IOMMU Group 0 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)

IOMMU Group 0 06:00.1 Audio device [0403]: NVIDIA Corporation GM200 High Definition Audio [10de:0fb0] (rev a1)

IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]

IOMMU Group 2 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)

IOMMU Group 2 07:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]

IOMMU Group 4 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]

IOMMU Group 4 08:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]

IOMMU Group 4 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]

IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]

IOMMU Group 5 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]

IOMMU Group 5 09:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

IOMMU Group 5 09:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]

IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)

IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)

IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]

IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]

IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]

IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]

IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]

IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]

IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]

IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]

```

Groups are not different from the other ones, :Sad: 

----------

## Myu

I'm not sure if I can help much more since it's becoming increasingly more hardware specific but there's someone with the exact same MB from the VFIO subreddit which managed to get it working apparently, maybe you can poke him/her ? : https://www.reddit.com/r/VFIO/comments/7ajpj7/cant_get_passthrough_to_work/

You can also ask directly there, there's quite a community with Ryzen boards : https://reddit.com/r/vfio

----------

## dylanmc

Thanks.

I have read/write somethin (but I don't have understand how I have to format code in that forum) now I have to wait.

I have understood that my group 2 is passable for passtrought, but I have to boot in to the secondary VAGa card, as written here https://www.reddit.com/r/VFIO/comments/616xih/gpu_passthrough_with_msi_b350_tomahawk/

but I didn't get it works

I pass to the kernel the parameter 

```
 Modify /etc/default/grub and add “vfio-pci.ids=dead:beef video=efifb:off” to GRUB_CMDLINE_LINUX_DEFAULT (use the PCI vid/pid of your primary GPU, not dead:beef)
```

 but nothing happens.

I use lilo, but the kernel parms are the same.. the primary card still boots up.

I think that I'm near the solution, but I miss something :/

----------

## dylanmc

Some good news here.

I was able to switch in the secondary VGA card, after booting up.

The system now goes in the secondart GPU.

I've passtrought the second GPU and the HDMI audio, and the emulator started, removese the messages from the previus boot from firt monitor/first GPU but eaves  monitor without signal  :Sad: 

And I don't understand why :/

The passtrougt GPU is under vfio-pci module.. damn .. I'm near but stalled for now. :/

----------

## Myu

We're slowly getting there  :Smile:  I see on Reddit that someone with the same MB managed to do it so there's hope !

I'm not sure I understood you last sentence though

So you isolate the 980Ti properly and are booted into Gentoo with you 710 right ?

Then you start QEMU and what happens ? You don't have signal on the screen output wired to the 980 ? Not even BIOS logo (Tianocore)

Do you still signal from you Gentoo desktop  while doing that ? Or does it freeze/crash ?

----------

## dylanmc

 *Myu wrote:*   

> We're slowly getting there  I see on Reddit that someone with the same MB managed to do it so there's hope !
> 
> I'm not sure I understood you last sentence though
> 
> So you isolate the 980Ti properly and are booted into Gentoo with you 710 right ?
> ...

 

I got the boot with the primary vga, the 980 ti.

During the boot, I've done 

```
#Section "Device"

#    BusID          "07:00.0"

#    Identifier     "Device0"

#    Driver         "nvidia"

#    VendorName     "NVIDIA Corporation"

#    BoardName      "GeForce GTX 760"

#EndSection

Section "Device"

    BusID          "06:00:0"

    Identifier     "Device1"

    Driver         "nvidia"

    VendorName     "NVIDIA Corporation"

    BoardName      "GeForce GT 710"

EndSection

```

to prevent the x server loads on primary GPU (908).

X server is running now on the secondary GPU (710).

I've launched the virt-manager, added the  isolated primary GPU (980) successfully ( I hope, no error messages as before) and the monitor who get the signal from the firts GPU seems to load someting, but after 2/3 seconds get black.

Maybee I have to isolate the primary GPU after he gets out from xorg.conf, but I've tried in kernel with no luck, tha card still shows me the first boot messages

The emulator not crash, I was ablòe also to passtrought keyboard and mouse, but I don't see anithing  :Sad: 

Maybee I have to pass the rom file.. I don't know.. or I must have to override firts gpu in the kernel.

(sorry for my english... I think is not so clear as your's  :Smile:  )

----------

## dylanmc

https://www.reddit.com/r/VFIO/comments/7mpuup/asus_b350_prime_plus_and_passtrought_problems/

fingers crossed..  :Smile: 

----------

## dylanmc

I'm not able to solve my problems..  I've read a lot of threads, with no luck.

I'm sill very close, the GPU Passtrought works, but the virt-manager launch a black screen  :Sad: 

----------

## szatox

 *Quote:*   

> I'm sill very close, the GPU Passtrought works, but the virt-manager launch a black screen

 Wait, what? 

Do I get you right? Are you trying to loop the output from a physical device back to a virtual console?

Try this instead: You just hook up another display (or another input of your display) to the video card you passed to your guest and launch that VM again.

----------

## keet

I'm not sure whether I'll be able to help at all, but I do have V.G.A. passthrough working on both Linux and Windows guests using plain qemu without virt-manager (the host is headless).  The configurations do vary on several lines, not out of necessity, but simply because I worked on optimizing/cleaning the Gentoo configuration more recently than the Windows one (which I haven't booted into in over a week... but performance was great when I did last use it).  Thus, please don't interpret differences between the two as necessary or even useful.  I've just been lazy and not used/reconfigured Windows recently.  For example, Gentoo uses U.E.F.I., but Windows uses B.I.O.S.  Gentoo originally used B.I.O.S., but I converted it, and I hope to do the same with Windows, once I actually want to use it again.

Gentoo:

```
exec qemu-system-x86_64 -enable-kvm \

   -drive file=/home/myuseraccount/OVMF.fd,if=pflash,format=raw,unit=0,readonly=off \

   -drive if=pflash,format=raw,file=/home/myuseraccount/OVMF_VARS.fd,readonly=off \

   -cpu host,kvm=off -smp 8 \

       -drive file=/home/myuseraccount/Gentoo.img,if=virtio \

   -m 8G \

   -M q35 \

   -netdev bridge,id=mynet1 -device virtio-net-pci,netdev=mynet1,id=nic2,mac=(M.A.C. address) \

        -name "Gentoo" \

   -device vfio-pci,host=01:00.0,x-vga=on \

   -device vfio-pci,host=01:00.1,bus=pcie.0 \

   -device nec-usb-xhci,id=xhci \

   -device usb-host,bus=xhci.0,vendorid=0x13fe,productid=0x5500 \

        -device usb-host,vendorid=0x04d9,productid=0x2013 \

   # many more lines like this for other U.S.B. devices \

   -vga none -nographic \

        $@
```

Windows 10:

```
exec qemu-system-x86_64 -enable-kvm \

   -cpu host,hv_vendor_id=Nvidia43FIX,kvm=off -smp sockets=1,cores=3,threads=2 \

       -drive file=/home/myuseraccount/Windows10.img,if=virtio \

   -m 8G -mem-prealloc \

   -machine type=q35,accel=kvm \

   -device ich9-usb-ehci1,id=usb \

   -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,multifunction=on \

   -device ich9-usb-uhci2,masterbus=usb.0,firstport=2 \

   -device ich9-usb-uhci3,masterbus=usb.0,firstport=4 \

   -soundhw hda \

   -netdev bridge,id=mynet1 -device virtio-net-pci,netdev=mynet1,id=nic2,mac=(M.A.C. address) \

        -name "Windows 10" \

   -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \

   -bios /usr/share/qemu/bios.bin \

   -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

   -device vfio-pci,host=01:00.1,bus=pcie.0 \

        -device usb-host,vendorid=04d9,productid=2013 \

   # Many more lines like this for other U.S.B. devices \

   -vga none -nographic \

        $@
```

My Gentoo configuration is a bit different than the Windows one because I simplified it a bit, but it did work when it was configured similarly to the Windows 10 one.

I also need to run these commands to unbind the devices and enable them for vfio-pci:

```
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind

echo 0000:01:00.1 > /sys/bus/pci/devices/0000:01:00.1/driver/unbind

echo 10de 1b81 > /sys/bus/pci/drivers/vfio-pci/new_id

echo 10de 10f0 > /sys/bus/pci/drivers/vfio-pci/new_id
```

Also:

```
cat /etc/udev/rules.d/10-qemu-hw-users.rules 

SUBSYSTEM=="vfio", OWNER="root", GROUP="kvm", MODE="0770"

KERNEL=="vfio", SUBSYSTEM=="misc", OWNER="root", GROUP="kvm", MODE="0660"

KERNEL=="1", SUBSYSTEM=="vfio", OWNER="root", GROUP="kvm", MODE="0660"

# chmod ug+rX /dev/vfio
```

Also, the network bridging was a bit tricky to set up, but I think that I used these commands:

```
ip tuntap add dev tap0 mode tap group kvm 

ip link set dev tap0 up promisc on 

ip addr add 0.0.0.0 dev tap0 

ip link add br0 type bridge 

ip link set br0 up 

ip link set tap0 master br0 

echo 0 > /sys/class/net/br0/bridge/stp_state 

ip addr add 10.0.1.1/24 dev br0 

sysctl net.ipv4.conf.tap0.proxy_arp=1 

sysctl net.ipv4.conf.eth1.proxy_arp=1 

sysctl net.ipv4.ip_forward=1 

iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE 

iptables -A FORWARD -i br0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT 

iptables -A FORWARD -i eth1 -o br0 -j ACCEPT
```

I mostly used these guides:

https://wiki.debian.org/VGAPassthrough

https://wiki.gentoo.org/wiki/QEMU (along with the 'usage' subpages)

https://www.evonide.com/non-root-gpu-passthrough-setup/

I'm sorry if this is excessive information, but I hope that it helps.[/url]

----------

## Myu

 *Quote:*   

> Wait, what?
> 
> Do I get you right? Are you trying to loop the output from a physical device back to a virtual console?
> 
> Try this instead: You just hook up another display (or another input of your display) to the video card you passed to your guest and launch that VM again.

 

Yeah +1 to what szatox said ! You won't get any output on the virt-manager Window, the output will be to the display cable plugged to the GPU you're passing through ! 

Try another screen or if you have a monitor with some free VGA / DVI / HDMI, hook a cable from one of those from the card

For instance, I have 

```
Motheboard HDMI -> Display HDMI1

Nvidia DVI -> Display HDMI2
```

And when booting the VM through virt-manager, I switch inputs from the screen OSD

----------

## dylanmc

No, there is a misunderstanding.

I have a monitor capable of two HDMI input, and when I launch the virtual machine, I didn't get any output in the HDMI where the gpu passtrought card is connected.

I only see it's doing something cause the init messages before the xorg switch to the secondary GPU disappears.

The onmly think I haqve to do is try to dump my GPU 980ti rom, but actually I didn't have any windows to grab it.

I've grabbed on linux, no luck on that..

----------

## dylanmc

Thanks keet, I will try your suggestions as soon as possible.

I ask you one thing: you boot on your secondary GPU or on the first GPU?

Because I boot on first GPU, the xorg switch to the secondary, ad X starts on it.

The first gpu gets the vfio module but it seems it's missing the rom or something else.[/quote]

----------

## keet

 *dylanmc wrote:*   

> I ask you one thing: you boot on your secondary GPU or on the first GPU?

 

The host uses the Intel graphics.  I never included the drivers for my NVidia card on the host, so it never tries to use it.

----------

## szatox

 *Quote:*   

> 
> 
> I only see it's doing something cause the init messages before the xorg switch to the secondary GPU disappears

 Ok, just to make sure, you see those init messages on your display connected to that passed-through video card, or on some virtual console connected somewhere else?

Do you have any control over that VM after it boots? Like, can you SSH into it to look around?

----------

## dylanmc

 *szatox wrote:*   

>  *Quote:*   
> 
> I only see it's doing something cause the init messages before the xorg switch to the secondary GPU disappears Ok, just to make sure, you see those init messages on your display connected to that passed-through video card, or on some virtual console connected somewhere else?
> 
> Do you have any control over that VM after it boots? Like, can you SSH into it to look around?

 

I've installed a ubuntu virtual machine, I have enabled ssh and works.

After, I've removed video qxl and tried to pass trought the GPU.

After boot, virt manager says the ther machine is running, but the ssh connection hangs and I can't connect.

So, I think there is another problem

----------

## dylanmc

Some news

I have installed an Ubuntu, in a WM.

Using a rom dumped by me or downloaded from a websiste, I got X finally working on the passtrought card, but it's still unusable, the ubuntu system won't boot.

The Windows one, give me no sign of life.

no errors in dmesg, no errors on qemu log

----------

## dylanmc

After TONS of configurations, I got my ubuntu works in gpu passtrought!!!

But the windows side, doesen't  :Sad: 

This is my configuration

```

snoopx /etc/libvirt/qemu # cat win10-2.xml 

<!--

WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE

OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:

  virsh edit win10-2

or other application using the libvirt API.

-->

<domain type='kvm'>

  <name>win10-2</name>

  <uuid>a803f169-9c79-4d86-bd24-fc7efdc43301</uuid>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>4194304</currentMemory>

  <vcpu placement='static'>10</vcpu>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>

  </os>

  <features>

    <acpi/>

    <apic/>

    <hyperv>

      <relaxed state='on'/>

      <vapic state='on'/>

      <spinlocks state='on' retries='8191'/>

      <vendor_id state='on' value='1234567890ab'/>

    </hyperv>

    <kvm>

      <hidden state='on'/>

    </kvm>

    <vmport state='off'/>

  </features>

  <cpu mode='host-model' check='partial'>

    <model fallback='allow'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

    <timer name='hypervclock' present='yes'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>destroy</on_crash>

  <pm>

    <suspend-to-mem enabled='no'/>

    <suspend-to-disk enabled='no'/>

  </pm>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/var/lib/libvirt/images/win10.qcow2'/>

      <target dev='sda' bus='sata'/>

      <boot order='1'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/var/lib/libvirt/images/win10-2.qcow2'/>

      <target dev='sdb' bus='sata'/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <source file='/home/luca/magazzino/sistemi_operativi/windows/it_windows_10_multiple_editions_version_1703_updated_march_2017_x64_dvd_10204062.iso'/>

      <target dev='hdb' bus='ide'/>

      <readonly/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci2'>

      <master startport='2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci3'>

      <master startport='4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </controller>

    <controller type='sata' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:23:75:59'/>

      <source bridge='br0'/>

      <model type='rtl8139'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <input type='tablet' bus='usb'>

      <address type='usb' bus='0' port='1'/>

    </input>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <graphics type='spice' autoport='yes'>

      <listen type='address'/>

      <gl enable='no' rendernode=''/>

    </graphics>

    <sound model='ich6'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </sound>

    <video>

      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>

    </video>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <source>

        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <source>

        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

    </hostdev>

    <redirdev bus='usb' type='spicevmc'>

      <address type='usb' bus='0' port='2'/>

    </redirdev>

    <redirdev bus='usb' type='spicevmc'>

      <address type='usb' bus='0' port='3'/>

    </redirdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </memballoon>

  </devices>

</domain>

```

----------

## dylanmc

A tons of try.. still passtrought only on a ubuntu  :Sad: 

----------

## dylanmc

I finally have installed the nvidia drivers in win10, bit I got error 43.

I've read a lot of forums and messages, but the card is still blocked.

I've tried to hide, but with no luck

----------

## keet

Do you have (at least the equivalent of):

-cpu host,kvm=off

----------

## dylanmc

```
luca@snoopx ~ $ sudo cat /etc/libvirt/qemu/dieci.xml 

<!--

WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE

OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:

  virsh edit dieci

or other application using the libvirt API.

-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

  <name>dieci</name>

  <uuid>a0a617cd-1340-4836-a110-ff567f1f1fb3</uuid>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>4194304</currentMemory>

  <vcpu placement='static'>8</vcpu>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

    <hyperv>

      <relaxed state='on'/>

      <vapic state='on'/>

      <spinlocks state='on' retries='8191'/>

      <vendor_id state='on' value='0123456789ab'/>

    </hyperv>

    <kvm>

      <hidden state='on'/>

    </kvm>

    <vmport state='off'/>

  </features>

  <cpu mode='host-passthrough' check='none'>

    <topology sockets='1' cores='4' threads='2'/>

    <feature policy='disable' name='smep'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

    <timer name='hypervclock' present='yes'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>destroy</on_crash>

  <pm>

    <suspend-to-mem enabled='no'/>

    <suspend-to-disk enabled='no'/>

  </pm>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/var/lib/libvirt/images/win10.qcow2'/>

      <target dev='hda' bus='ide'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci2'>

      <master startport='2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci3'>

      <master startport='4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:91:09:f6'/>

      <source bridge='br0'/>

      <model type='rtl8139'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='spicevmc'>

      <target type='virtio' name='com.redhat.spice.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <input type='tablet' bus='usb'>

      <address type='usb' bus='0' port='1'/>

    </input>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <graphics type='spice' autoport='yes'>

      <listen type='address'/>

      <gl enable='no' rendernode=''/>

    </graphics>

    <sound model='ich6'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </sound>

    <video>

      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </video>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <source>

        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <source>

        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>

      </source>

      <rom bar='off'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>

    </hostdev>

    <redirdev bus='usb' type='spicevmc'>

      <address type='usb' bus='0' port='2'/>

    </redirdev>

    <redirdev bus='usb' type='spicevmc'>

      <address type='usb' bus='0' port='3'/>

    </redirdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </memballoon>

  </devices>

  <qemu:commandline>

    <qemu:arg value='-cpu'/>

    <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>

  </qemu:commandline>

</domain>

```

I have only to try to install the GPU to isolate in the second slot, nothing else is working..

Or get a Radeon instead of Nvidia..

----------

## keet

Are you trying to use HyperV inside the guest?  If not, I would remove that section.

It looks like you have several virtual U.S.B. controllers.  They're probably unrelated, but they might also be unnecessary.  I just pass through my host's U.S.B. controller and plug a hub with all my peripherals into one of its ports.

The same (regarding it being unnecessary) might be true of the P.C.I. controller, serial port, and console.

I wonder whether the Spice console might be causing problems; can you try removing the Spice adapter along with qxl video, the U.S.B. tablet device, mouse, keyboard, and other Spice devices/configurations?

I wonder whether specifying a null vendor identifier is causing problems; it might be worth removing that.  I'm not sure what the rest of the Qemu command line does.  It looks like you have the '-cpu' switch but without any arguments, unless it counts 'host' on the next line.

I don't know what you need, but it seems like your configuration file might have many unnecessary things.  Trimming them away might fix the problem or at least make it easier to diagnose.

----------

## dylanmc

Are you successfully pass an nvidia card to windows?

I removed hyperv, same error

Removed some usb stuff as you suggest, still get the same error.

If I remove the spice console I didn't see anything.

The nvidia driver still detects kvm and gets me error 43..

----------

## keet

Yes, I successfully passed through my GTX 1070 to Windows 10 (as well as Gentoo and Debian Linux) and tested it successfully in 3D games.  This is my command:

```
exec qemu-system-x86_64 -enable-kvm \

   -cpu host,kvm=off -smp sockets=1,cores=2,threads=2 \

   -m 8G \

   -bios /usr/share/qemu/bios.bin \

   -drive file=/home/myuser/Windows10.img,if=virtio \

   -netdev bridge,id=mynet1 -device virtio-net-pci,netdev=mynet1,id=nic2,mac=mymacaddress \

   -name "Windows 10" \

   -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \

   -device vfio-pci,host=03:00.0 \

   -vga none -nographic \

   "$@"
```

I no longer pass through the graphic's sound because it doesn't work properly in Gentoo and because I don't really need it.  However, it didn't cause any problems when it was included.  Apart from that, I recently removed other things that I copied or adapted from various online guides, not because they prevented it from working, but because they seem unnecessary (it works fine without them), so that the command is much shorter.

----------

## bunder

 *Myu wrote:*   

>  *Quote:*   Wait, what?
> 
> Do I get you right? Are you trying to loop the output from a physical device back to a virtual console?
> 
> Try this instead: You just hook up another display (or another input of your display) to the video card you passed to your guest and launch that VM again. 
> ...

 

actually, wendell from level1techs got that working recently.  the way he got it set up, you don't need a monitor on the guest GPU, it can be sent back to spice and acceleration apparently works too.

----------

## keet

That is true, but I think that it's more complicated to set up, so it might be better to focus on making passthrough work first, and then work on passthrough in Spice later.  I was pasting my configuration because it's (at least close to) the minimum that one needs to make it work.    If we simplify it as much as possible first, then we can work on adding features later.

I'm running mine on a server with Intel and NVidia graphics.  The NVidia graphics card is never used by the host -- it has no kernel configuration for it and no NVidia drivers installed.  I have only one monitor, which is connected to the Intel (motherboard) graphics and the video card.  However, the host never displays anything through the discrete graphics card; only virtual machines do that.

I don't know about the benefits of using virt-manager, but it seems like it might be adding quite a few unnecessary things here.  At the very least, it might help to strip them until the G.P.U. is working, and then add them back and see whether it still works.  HyperV is another hypervisor, so enabling support for it might complicate matters, even if it's only running HyperV as a host inside the guest (nested virtualization).  I don't know how much is necessary to make a V.M. work with virt-manager, but it looks to me like the guest's U.S.B. controllers, serial ports, tablet input device, keyboard, mouse, sound, memory ballooning, P.C.I. and I.D.E. controllers, and Spice support could all be removed.  You might not see anything with Spice disabled, but in that case, it might be best to enable Remote Desktop and use rdesktop or something similar to shut it down if it doesn't work.  Many people here probably know virt-manager much better than I do and can correct me.

The one other thing that I would check is the syntax of the <qemu:commandline> section.  'kvm=off' should take care of the problem for you, and I do see it there, but maybe it's not working because of syntax.  Can anyone here comment on the syntax of that section?  Compared to my command line, it has a couple extra parameters.

----------

## dylanmc

I've else tried to change my GPU Nvidia card, an 1060 GTX: but the same error, 43.

I think that passing a grabbed rom can be useful, but I'm unable to grab it correctly

----------

## dylanmc

Finally, after a lot of tryes, I got my GPU passtrought work.

I had to change my card from Nvidia to AMD.. the only thing that was blocking me was the NVIDIA Win10 drivers :/

Bye bye Nvidia !!!

----------

## Anon-E-moose

Definitely easier to passthrough amd cards, don't know why nvidia makes it so hard, but they do.

----------

## dylanmc

I think the only want to sell' quadros.

No other reasons forum me..

----------

## dylanmc

another question : does anyone passtrought an entire partition to the guest system?

----------

## Belliash

Hello,

I own a notebook - Clevo P170SM with i7-4800mq and gtx770m installed.

I have managed to run Windows 10 in KVM and successfully passed the nvidia card with vfio. Unfortunately as this is a laptop, there is no way to connect external monitor. Thus I decided to give it a try with VNC sever.

I installed Heaven benchmark on both VM and bare metal and compared FPS. On bare metal I get around 100FPS, while in VM it reports just around 30-40FPS and video is very laggy. I suppose that lag can be related to VNC but why I got just up to 40FPS?

----------

