# Lots of errors with Xen

## Thoth93

This is my first time using Gentoo. I'm trying to setup a Gentoo system with uefi, systemd, mirrored btrfs for main system and zfs for xen VMs

So far everything has worked out fine, Im actualy surprised how easy it was after a bit of searching wiki, forums and taking my own notes

to set this up without any errors or problems, Except for one peace of the puzzle Xen. First grub would not generate the menu for it based on both

of these problems at https://wiki.gentoo.org/wiki/Talk:Xen under "GRUB2" ** then once that was done the file /etc/grub.d/20_linux_xen had to be

manualy edited before I ran grub-mkconfig because otherwise grub wont identify your system as uefi and add the right options, without this your

system will reboot when you try to boot into xen ***. So now it boots but there are a lot of errors including usb errors so I cannot login either.

I then setup sshd so I could get journal errors here they are, If anyone needs more info just let me know. I've set up xen before on differnent

distro so I know it works with my system, also both 4.4.39 and 4.9.6-r1 work fine when not booted into xen.

Any one have Xen 4.7.1, 4.7 etc. running on gentoo-sources kernel-4.4.39, or the new 4.9.6-r1 ?

Also USE flags for app-emulation/xen, efi : Adds efi boot support, requires LDFLAG -melf_x86_64 for amd64

what happens if this is not set? is it still needed? This was not set when it was emerged. Not sure if it's relevant

**Fix: edit /etc/portage/make.conf

```
GRUB_PLATFORMS="efi-64 xen"
```

Note: their might be another way around the uefi/grub problem, add xen.efi directly to your os boot list

with sys-boot/efibootmgr? . . .

http://xenbits.xenproject.org/docs/unstable/misc/efi.html

Compile grub with "efi-64 xen"

Edit /etc/grub/20_linux_xen (line 126) and

add "efi" as a recognized boot platform:

***Fix: https://lists.gt.net/xen/users/433990?page=last

```

---- Lines 125 - 30:

echo '$(echo "$xmessage" | grub_quote)'

if [. "\$grub_platform" = "pc" -o "\$grub_platform" = "efi" -o

"\$grub_platform" = "" ]; then

xen_rm_opts=

else

xen_rm_opts="no-real-mode edd=off"

fi

---- 

```

Info:

/etc/portage/package.use

```

app-emulation/xen efi

app-emulation/xen-tools hvm ovmf pygrub python qemu screen

# required by app-emulation/xen-tools-4.7.1-r4::gentoo[ovmf,python_targets_python2_7]

# required by app-emulation/xen-4.7.1-r4::gentoo

# required by app-emulation/xen (argument)

>=dev-lang/python-2.7.12:2.7 sqlite

# required by app-emulation/xen-4.7.1-r4::gentoo[efi]

# required by app-emulation/xen (argument)

>=sys-devel/binutils-2.25.1-r1 multitarget

```

I do not know what the errors mean or how to fix it, so far the only thing I've tried is to update to

the new 4.9.6-r1 kernel. I get the same errors on 4.9.6-r1 as with 4.4.39

uname

```

user@ Hostname #> uname -a

Linux Hostname 4.9.6-gentoo-r1 #1 SMP Sun Feb 26 15:43:52 EST 2017 x86_64 Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz GenuineIntel GNU/Linux

```

xl dmesg . . . seems ok ? except for "ACPI Error (tbxfroot-0218): A valid RSDP was not found"

```

user@ Hostname #> xl dmesg

 Xen 4.7.1

(XEN) [    0.000000] Xen version 4.7.1 (@[unknown]) (x86_64-pc-linux-gnu-gcc (Gentoo 4.9.4 p1.0, pie-0.6.4) 4.9.4) debug=n Sat Feb 25 13:45:51 EST 2017

(XEN) [    0.000000] Latest ChangeSet: 

(XEN) [    0.000000] Bootloader: GRUB 2.02~beta3

(XEN) [    0.000000] Command line: placeholder console_timestamps=boot dom0_mem=4056M,max:8192M noreboot

(XEN) [    0.000000] Video information:

(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16

(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds

(XEN) [    0.000000] Disc information:

(XEN) [    0.000000]  Found 0 MBR signatures

(XEN) [    0.000000]  Found 0 EDD information structures

(XEN) [    0.000000] Multiboot-e820 RAM map:

(XEN) [    0.000000]  0000000000000000 - 00000000000a0000 (usable)

(XEN) [    0.000000]  0000000000100000 - 0000000078f7d000 (usable)

(XEN) [    0.000000]  0000000078f7d000 - 0000000079787000 (reserved)

(XEN) [    0.000000]  0000000079787000 - 00000000798c0000 (usable)

(XEN) [    0.000000]  00000000798c0000 - 0000000079d4c000 (ACPI NVS)

(XEN) [    0.000000]  0000000079d4c000 - 000000007bca1000 (reserved)

(XEN) [    0.000000]  000000007bca1000 - 000000007bd02000 type 20

(XEN) [    0.000000]  000000007bd02000 - 000000007bd03000 (usable)

(XEN) [    0.000000]  000000007bd03000 - 000000007bd89000 (reserved)

(XEN) [    0.000000]  000000007bd89000 - 000000007c000000 (usable)

(XEN) [    0.000000]  000000007c000000 - 0000000090000000 (reserved)

(XEN) [    0.000000]  00000000fed1c000 - 00000000fed45000 (reserved)

(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)

(XEN) [    0.000000]  0000000100000000 - 0000000480000000 (usable)

(XEN) [    0.000000] ACPI Error (tbxfroot-0218): A valid RSDP was not found [20070126]

(XEN) [    0.000000] System RAM: 16274MB (16665400kB)

(XEN) [    0.000000] Domain heap initialised

(XEN) [    0.000000] Processor #00 6:15 APIC version 21

(XEN) [    0.000000] Processor #02 6:15 APIC version 21

(XEN) [    0.000000] Processor #04 6:15 APIC version 21

(XEN) [    0.000000] Processor #06 6:15 APIC version 21

(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 2 I/O APICs

(XEN) [    0.000000] Not enabling x2APIC: depends on iommu_supports_eim.

(XEN) [    0.000000] xstate_init: using cntxt_size: 0x340 and states: 0x7

(XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU0 bank 17, using 0x1

(XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU0 bank 18, using 0x1

(XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU0 bank 19, using 0x1

(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)

(XEN) [    0.968018] Detected 3500.075 MHz processor.

(XEN) [    0.972677] Initing memory sharing.

(XEN) [    0.973851] I/O virtualisation disabled

(XEN) [    0.975002] Enabled directed EOI with ioapic_ack_old on!

(XEN) [    0.976560] ENABLING IO-APIC IRQs

(XEN) [    0.977705]  -> Using old ACK method

(XEN) [    1.576984] Platform timer is 1.193MHz PIT

(XEN) [    1.578159] Allocated console ring of 16 KiB.

(XEN) [    1.579325] VMX: Supported advanced features:

(XEN) [    1.580479]  - APIC MMIO access virtualisation

(XEN) [    1.581637]  - APIC TPR shadow

(XEN) [    1.582782]  - Extended Page Tables (EPT)

(XEN) [    1.583936]  - Virtual-Processor Identifiers (VPID)

(XEN) [    1.585098]  - Virtual NMI

(XEN) [    1.586240]  - MSR direct-access bitmap

(XEN) [    1.587392]  - Unrestricted Guest

(XEN) [    1.588557]  - APIC Register Virtualization

(XEN) [    1.589710]  - Virtual Interrupt Delivery

(XEN) [    1.590864]  - Posted Interrupt Processing

(XEN) [    1.592019]  - VMCS shadowing

(XEN) [    1.593164]  - VM Functions

(XEN) [    1.594308] HVM: ASIDs enabled.

(XEN) [    1.595454] HVM: VMX enabled

(XEN) [    1.596597] HVM: Hardware Assisted Paging (HAP) detected

(XEN) [    1.597762] HVM: HAP page sizes: 4kB, 2MB, 1GB

(XEN) [    1.659907] Brought up 4 CPUs

(XEN) [    1.684096] Dom0 has maximum 624 PIRQs

(XEN) [    1.685300] ACPI is disabled, notifying Domain 0 (acpi=off)

(XEN) [    1.686468] *** LOADING DOMAIN 0 ***

(XEN) [    1.801353]  Xen  kernel: 64-bit, lsb, compat32

(XEN) [    1.802516]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c3c000

(XEN) [    1.804919] PHYSICAL MEMORY ARRANGEMENT:

(XEN) [    1.806079]  Dom0 alloc.:   0000000470000000->0000000472000000 (1029028 pages to be allocated)

(XEN) [    1.808413]  Init. ramdisk: 000000047fba4000->000000047ffff3e0

(XEN) [    1.809600] VIRTUAL MEMORY ARRANGEMENT:

(XEN) [    1.810758]  Loaded kernel: ffffffff81000000->ffffffff81c3c000

(XEN) [    1.811931]  Init. ramdisk: 0000000000000000->0000000000000000

(XEN) [    1.813101]  Phys-Mach map: 0000008000000000->00000080007ec000

(XEN) [    1.814272]  Start info:    ffffffff81c3c000->ffffffff81c3c4b4

(XEN) [    1.815441]  Page tables:   ffffffff81c3d000->ffffffff81c50000

(XEN) [    1.816611]  Boot stack:    ffffffff81c50000->ffffffff81c51000

(XEN) [    1.817782]  TOTAL:         ffffffff80000000->ffffffff82000000

(XEN) [    1.818969]  ENTRY ADDRESS: ffffffff81aaa180

(XEN) [    1.820481] Dom0 has maximum 4 VCPUs

(XEN) [    1.896830] Scrubbing Free RAM on 1 nodes using 4 CPUs

(XEN) [    1.923470] ....................................done.

(XEN) [    2.694688] Initial low memory virq threshold set at 0x4000 pages.

(XEN) [    2.695856] Std. Loglevel: Errors and warnings

(XEN) [    2.697005] Guest Loglevel: Nothing (Rate-limited: Errors and warnings)

(XEN) [    2.698186] Xen is relinquishing VGA console.

(XEN) [    2.700103] *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)

(XEN) [    2.700148] Freed 308kB init memory

```

Journalctl, If you want dmesg or the whole journalctl --boot just ask

```

user@ Hostname #> journalctl -p 4 -xb

Feb 27 16:07:48 Hostname kernel: ------------[ cut here ]------------

Feb 27 16:07:48 Hostname kernel: WARNING: CPU: 0 PID: 1 at mm/slab_common.c:861 kmalloc_slab+0x39/0x8b

Feb 27 16:07:48 Hostname kernel: Modules linked in:

Feb 27 16:07:48 Hostname kernel: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.9.6-gentoo-r1 #1

Feb 27 16:07:48 Hostname kernel: Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0 12/17/2015

Feb 27 16:07:48 Hostname kernel:  ffffc9000224fde8 ffffffff8131f65b 0000000000000000 ffffffff8192ffd9

Feb 27 16:07:48 Hostname kernel:  ffffc9000224fe28 ffffffff810804a0 0000035d0224fd50 0000000000000000

Feb 27 16:07:48 Hostname kernel:  00000000024080c0 0000000000000000 0000000000000000 0000000000000000

Feb 27 16:07:48 Hostname kernel: Call Trace:

Feb 27 16:07:48 Hostname kernel:  [<ffffffff8131f65b>] dump_stack+0x61/0x7e

Feb 27 16:07:48 Hostname kernel:  [<ffffffff810804a0>] __warn+0xc0/0xdb

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81080568>] warn_slowpath_null+0x18/0x1a

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81121377>] kmalloc_slab+0x39/0x8b

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81140af1>] __kmalloc+0x11/0xfd

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81adf062>] ? get_max_acpi_id+0x7e/0x7e

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81adf0bb>] xen_acpi_processor_init+0x59/0x1af

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81adf062>] ? get_max_acpi_id+0x7e/0x7e

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81002177>] do_one_initcall+0x8e/0x11b

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81aab035>] kernel_init_freeable+0x163/0x1eb

Feb 27 16:07:48 Hostname kernel:  [<ffffffff815ff6ed>] ? rest_init+0x74/0x74

Feb 27 16:07:48 Hostname kernel:  [<ffffffff815ff6f6>] kernel_init+0x9/0xf8

Feb 27 16:07:48 Hostname kernel:  [<ffffffff81605e45>] ret_from_fork+0x25/0x30

Feb 27 16:07:48 Hostname kernel: ---[ end trace 8ee6fe0b1ffb748c ]---

Feb 27 16:07:48 Hostname kernel: genirq: Flags mismatch irq 1. 00000080 (i8042) vs. 0002cc00 (spinlock0)

Feb 27 16:07:48 Hostname kernel: i8042: probe of i8042 failed with error -16

Feb 27 16:07:48 Hostname kernel: genirq: Flags mismatch irq 8. 00000080 (rtc0) vs. 00000000 (xen-pcpu)

Feb 27 16:07:48 Hostname kernel: rtc_cmos: probe of rtc_cmos failed with error -16

Feb 27 16:07:48 Hostname kernel: mce: Unable to init device /dev/mcelog (rc: -16)

Feb 27 16:07:48 Hostname kernel: ata1.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata1.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata1.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata2.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata2.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata2.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata3.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata3.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata3.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata4.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata4.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: ata4.00: qc timeout (cmd 0xec)

Feb 27 16:07:48 Hostname kernel: ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Feb 27 16:07:48 Hostname kernel: spl: loading out-of-tree module taints kernel.

Feb 27 16:07:48 Hostname kernel: znvpair: module license 'CDDL' taints kernel.

Feb 27 16:07:48 Hostname kernel: Disabling lock debugging due to kernel taint

Feb 27 16:07:48 Hostname systemd[1]: var-lib-xenstored.mount: Where= setting doesn't match unit name. Refusing.

Feb 27 16:07:48 Hostname systemd[1]: xenstored.socket: Cannot add dependency job, ignoring: Unit var-lib-xenstored.mount failed to lo

Feb 27 16:07:48 Hostname systemd[1]: xenstored_ro.socket: Cannot add dependency job, ignoring: Unit var-lib-xenstored.mount failed to

Feb 27 16:07:48 Hostname systemd[1]: xenconsoled.service: Cannot add dependency job, ignoring: Unit var-lib-xenstored.mount failed to

Feb 27 16:07:48 Hostname systemd[1]: display-manager.service: Cannot add dependency job, ignoring: Unit display-manager.service faile

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'evtchn'

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'gntdev'

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'netbk'

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'blkbk'

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'xen-scsibk'

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'usbbk'

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'pciback'

Feb 27 16:07:48 Hostname systemd-modules-load[1826]: Failed to find module 'blktap2'

Feb 27 16:07:48 Hostname systemd[1]: Failed to start Load Kernel Modules.

-- Subject: Unit systemd-modules-load.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- Unit systemd-modules-load.service has failed.

-- 

-- The result is failed.

Feb 27 16:07:48 Hostname systemd[1]: systemd-modules-load.service: Failed with result 'exit-code'.

Feb 27 16:07:48 Hostname kernel: genirq: Flags mismatch irq 5. 00000080 (i801_smbus) vs. 0002cc00 (callfuncsingle0)

Feb 27 16:07:48 Hostname kernel: i801_smbus 0000:00:1f.3: Failed to allocate irq 5: -16

Feb 27 16:07:48 Hostname xenstored[1973]: Checking store ...

Feb 27 16:07:48 Hostname xenstored[1973]: Checking store complete.

Feb 27 16:07:48 Hostname xenstored[1973]: Checking store ...

Feb 27 16:07:48 Hostname xenstored[1973]: Checking store complete.

Feb 27 16:07:48 Hostname dhcpcd[1934]: no interfaces have a carrier

Feb 27 16:07:51 Hostname kernel: snd_hda_intel 0000:04:00.1: azx_get_response timeout, switching to polling mode: last cmd=0x000f0001

Feb 27 16:07:52 Hostname kernel: xhci_hcd 0000:00:14.0: Error while assigning device slot ID

Feb 27 16:07:52 Hostname kernel: xhci_hcd 0000:00:14.0: Max number of devices this xHCI host supports is 32.

Feb 27 16:07:52 Hostname kernel: usb usb1-port1: couldn't allocate usb_device

Feb 27 16:08:04 Hostname kernel: xhci_hcd 0000:00:14.0: Error while assigning device slot ID

Feb 27 16:08:04 Hostname kernel: xhci_hcd 0000:00:14.0: Max number of devices this xHCI host supports is 32.

Feb 27 16:08:04 Hostname kernel: usb usb1-port2: couldn't allocate usb_device

Feb 27 16:08:05 Hostname dhcpcd[2047]: eno1: no IPv6 Routers available

Feb 27 16:08:16 Hostname kernel: xhci_hcd 0000:00:14.0: Error while assigning device slot ID

Feb 27 16:08:16 Hostname kernel: xhci_hcd 0000:00:14.0: Max number of devices this xHCI host supports is 32.

Feb 27 16:08:16 Hostname kernel: usb usb1-port12: couldn't allocate usb_device

Feb 27 16:11:31 Hostname systemd[1]: xenstored.socket: Cannot add dependency job, ignoring: Unit var-lib-xenstored.mount failed to lo

Feb 27 16:11:31 Hostname systemd[1]: xenstored_ro.socket: Cannot add dependency job, ignoring: Unit var-lib-xenstored.mount failed to

Feb 27 16:11:31 Hostname systemd[2144]: Couldn't move remaining userspace processes, ignoring: Permission denied

```

If "Failed to find module" is a different problem all together I can repost for that. Any replies or help is greatly appreciated.

----------

## Thoth93

So decided  :Idea:  to try to boot in to Xen in bios mode not uefi. I updated grub to support i386-pc

made a bios bootable usb with grub2 on it. This works, it works on both kernels 4.4.39 and 4.9.6-r1 and

it works with X2APIC enabled in bios. Before it would not boot in UEFI with X2APIC enabled in the bios at all

it would stop at "x2APIC: already enabled in BIOS, but iommu_supports_eim failed". And if boot into xen

with uefi with X2APIC disabled it would boot but I would get the trace call, ata and xhci erros and couldn't

login due to no usb keyboard support.

The errors about modules failing to load are still unresolved. I'll repost for that

And ill repost now that I know more about my problem.

Does anyone have Xen on gentoo working with UEFI with any kernel ? Does gentoo support this at all ?

The wiki doesnt say its not supported. Right now the only way to get it to work is it format everything and

start over cause right now I have a UEFI+GPT setup and you need to make a grub_bios boot partition to do

bios boot on this setup. And even then still need UEFI as a requirement for this setup.

----------

