# QEMU/KVM unable to boot amd64 guests on amd64 host

## sylence

Hey,

I'm struggling with my QEMU/KVM setup. The Gentoo system in question is more than 10 years old and was continuously updated, this could very well be an issue of me having messed up something at some point. Hardware-wise I'm running on an Intel Core2Quad CPU.

My main issue is that QEMU with enabled KVM works fine with 32 bit guests, but is unable to boot anything that is built for 64 bits. I simplified the problem to simply invoking

```
qemu-system-x86_64 --enable-kvm -m 384 -boot d -cdrom debian-10.3.0-amd64-netinst.iso
```

This will start the VM, the boot menu will appear, but booting any 64bit linux kernel like that will simply reboot the virtual machine without any error message. I verified that by manually invoking the kernel from the GRUB2 shell:

```
linux /boot/.../vmlinuz

boot

```

will just lead to an immediate restart of the VM.

Playing around with various QEMU settings such as "-cpu host" didn't change that, however as soon as I use that switch to simulate a 32 bit CPU, e.g. "-cpu pentium3", any attempt to boot a linux kernel will hang with a message that I need a 64 bit CPU. Which leads me to believe that my regular VM is a 64bit one and something else is wrong here. Creating the VM without KVM support (without --enable-kvm) "just works" with all my 64 bit ISOs. Moreover, with enabled KVM, 32 bit guests work correctly as well. I tried various ISOs for different operating systems, but none of those 64 bit images work: Windows bluescreens with "IRQL_NOT_LESS_OR_EQUAL", Solaris just reboots without any errors.

I'm currently running QEMU 4.2.0 and kernel 4.19.82 with KVM support:

```
CONFIG_HAVE_KVM=y

CONFIG_HAVE_KVM_IRQCHIP=y

CONFIG_HAVE_KVM_IRQFD=y

CONFIG_HAVE_KVM_IRQ_ROUTING=y

CONFIG_HAVE_KVM_EVENTFD=y

CONFIG_KVM_MMIO=y

CONFIG_KVM_ASYNC_PF=y

CONFIG_HAVE_KVM_MSI=y

CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y

CONFIG_KVM_VFIO=y

CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y

CONFIG_KVM_COMPAT=y

CONFIG_HAVE_KVM_IRQ_BYPASS=y

CONFIG_KVM=m

CONFIG_KVM_INTEL=m
```

PS: 64bit guests in Virtualbox run fine with VT-x enabled.

Any help is greatly appreciated!

----------

## mike155

Hi sylence,

welcome to the Gentoo forums!

Is virtualization enabled in the BIOS? It is/was disabled by default, at least on older machines.

Mike

----------

## Hu

Are you sure that the 64-bit guest you are trying to run is actually compatible with your real CPU?  When using software emulation without KVM, qemu can emulate instructions that you do not have.  When using KVM acceleration, I am not sure if qemu will fall back to emulating instructions that are not natively supported by the CPU.

----------

## sylence

Hey,

 *mike155 wrote:*   

> Is virtualization enabled in the BIOS? It is/was disabled by default, at least on older machines.

 

Yes, I just verified that - it's enabled. If I understood the documentation correctly, KVM would complain in case it's disabled.

 *Hu wrote:*   

> Are you sure that the 64-bit guest you are trying to run is actually compatible with your real CPU?  When using software emulation without KVM, qemu can emulate instructions that you do not have.  When using KVM acceleration, I am not sure if qemu will fall back to emulating instructions that are not natively supported by the CPU.

 

Well, the ISOs I'm trying to boot are mostly just regular x86_64 Linux distributions such as the Debian installer or the grml live CD. The 64bit version of Windows 7 doesn't work as well (bluescreen - see my initial post). All of those should run fine on my Core2Quad CPU. The crashes and reboots happen with '-cpu host' (passthrough) as well as '-cpu qemu64' or any other 64 bit option in the built-in CPU list. However, a 32bit ISO boots correctly with any 32 bit CPU (such as '-cpu qemu32'), even with KVM enabled.

I'm really out of ideas here.  :Sad: 

----------

## NeddySeagoon

sylence,

Do you have a 64 bit install on the host?

Most virtualisers will allow you to run 64 bit guests on 32 bit hosts, but not all of them.

What does the hosts 

```
uname -a
```

 show.

----------

## sylence

Yeah, the host's Gentoo installation is 64 bit:

```
Linux shade 4.19.82-gentoo.shade.v2 #2 SMP Thu Apr 9 12:49:07 CEST 2020 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
```

----------

## NeddySeagoon

sylence,

Do you have

```
$ ls -l /dev/kvm

crw-rw---- 1 root kvm 10, 232 Dec 30  2017 /dev/kvm
```

and is the user running qemu-system-x86_64 in the kvm group?

----------

## sylence

Looks fine (the user running qemu is 'sylence'):

```
# ls -l /dev/kvm

crw-rw---- 1 root kvm 10, 232 Mai 10 15:19 /dev/kvm

# grep kvm /etc/group

kvm:x:78:qemu,sylence
```

----------

## NeddySeagoon

sylence,

That's fine if sylence has logged out and back in again since being added to the kvm group.

If kvm is not listed in groups, when run as sylence, thats a problem.

```
$ groups

tty wheel uucp audio cdrom video games kvm cdrw usb users vboxusers scanner wireshark plugdev roy
```

----------

## sylence

Looks correct as well.

```
# groups

wheel cdrom video games kvm usb users libvirt docker sylence plugdev qemu vboxusers wireshark
```

In the meantime, I removed the VirtualBox kernel modules I had installed as well, just to be sure: So far, same behaviour.

----------

## mike155

Looks good so far...

Please post the output of

```
emerge --info
```

Please post your kernel config using wgetpaste.

----------

## sylence

Kernel config

emerge --info

----------

## mike155

Thanks for the data. I couldn't find anything wrong in the output of 'emerge --info' and your kernel config.

You seem to build KVM as modules:

```
CONFIG_KVM=m

CONFIG_KVM_INTEL=m
```

Are all KVM modules loaded when you start QEMU/KVM? Have you tried to compile KVM directly into the kernel?

----------

## sylence

If by all you mean all two of them (kvm and kvm_intel), then yes, those are loaded when I'm starting QEMU.

I haven't tried yet to build them into the kernel and can try to do that, but would be surprised if that changes anything.

Thanks for having a look at my configuration, though.  :Smile: 

----------

## alamahant

Do you have libvirt also installed?

Maybe if you had a full qemu/libvirt/virt-manager setup and created a guest graphically?

Sometimes trying to be smartly minimal in many things ends up inviting more problems than not..........

I have seen this in USE flags for example, the kernel .config flags  and in many other places...

We strive for smart seek minimal setups(which is a fantastic thing),but some times we break things up like this.................

 :Smile: 

----------

## sylence

I do have libvirt and virt-manager installed. That's where I noticed that something is wrong - my original goal was to set up an Android VM, which didn't work. Only then I noticed that I can't boot anything that's x86_64. The minimal QEMU command above is just my attempt to reduce the problem to something tangible. Sadly, the 'normal' way of managing KVM VMs doesn't work either.  :Wink: 

----------

## NeddySeagoon

sylence,

What error do you get from virtmanager?

I'll guess, none at all, the guests just reboot.

That points the finger at the host.  I have an AMD and Intel system that are KVM hosts. I can share the kernel .configs if you want.

----------

## sylence

 *NeddySeagoon wrote:*   

> What error do you get from virtmanager?
> 
> I'll guess, none at all, the guests just reboot.

 

Correct, no error anywhere. The guest just reboots (or bluescreens in the case of a Windows guest). I was playing around with QEMUs logging options ("-d") for invalid opcodes (amongst others), but didn't get anything from there either. I'm not sure if another kernel config would help. Guess I'll try to two things now: Building a 5.x kernel (maybe it's really just a very specific kernel bug) and booting some other distribution to see whether KVM works from there. Could as well be a hardware issue with the VT-X extension.

----------

## mike155

You could also try a BIOS update. I found some reports from users with a Core 2 CPU who reported that virtualization worked for them only after a BIOS update. 

12 years ago, virtualization was quite new and mainboard and BIOS manufactures had to puzzle out how to use it...

----------

## ununu

greets,

this could be due to little RAM on your qemu options 

 *Quote:*   

> -m 384

 

I think 64 bit distros needs a little bit more due to KASLR or something like that.

Also make sure that qemu is built with:

```
QEMU_SOFTMMU_TARGETS="i386 x86_64"

QEMU_USER_TARGETS="i386 x86_64"
```

I forgot to add this:

```
-cpu core2duo
```

or

```
-cpu Nehalem
```

 *sylence wrote:*   

> Hey,
> 
> ...
> 
> system is more than 10 years old ... running on an Intel Core2Quad CPU.

 

:cough:AMD:cough: <---

----------

## sylence

Problem solved, even though I'm still not sure what exactly the problem was.

I tried various things, all of them without success:

* BIOS is up to date (I used VT-x a lot previously with VirtualBox, under Windows etc., so that should work)

* Not enough RAM: The problem persisted also with 2GB or 4GB of RAM (-m 2048, -m 4096)

* My QEMU_*_TARGETS are both set to just "x86_64", which should give me at the very minimum support for 64 bit

* Different "-cpu" settings didn't help

I verified that it's not a hardware problem by booting another USB live linux (also with kernel 4.19) and running QEMU/KVM from there (same command, same ISO) - this worked fine. As a last resort I finally upgraded my kernel to 5.4.38. I built that one with the exact same config as I used before for 4.19.x (make olddefconfig). Now it works fine, no idea what was wrong with the old one. I also tried to integrate KVM into the kernel or build it as a set of modules - no difference.

Well, thanks for all the suggestions!  :Smile: 

----------

