# Guest VM's failt to start under 2.6.39, 3.0 Kernel, Xen 4.1

## DingbatCA

Dom0 was running a 2.6.34 kernel and everything worked.  Installed a 2.6.39 kernel and Guest VM's (DomU) cant start.  Installed the new 3.0 kernel, same problem.

If I reboot and start Dom0 with my 2.6.34 kernel, everything works.

Ideas?

Errors

```

nas configs # xm create xenspark.cfg

Using config file "./xenspark.cfg".

Error: Device 51712 (vbd) could not be connected. Path closed or removed during hotplug add: backend/vbd/3/51712 state: 1

nas configs # xl create xenspark.cfg

Parsing config file xenspark.cfg

libxl: error: libxl_dm.c:827:libxl__create_device_model: device model /usr/lib/xen/bin/qemu-dm is not executable: No such file or directory

Daemon running with PID 2780

```

xenspark.cfg

```

kernel = "/data/VMs/kernels/linux-2.6.38-gentoo-r5"

extra = "root=/dev/xvda1 clock=pit nohpet nopmtimer hpet=disable fastboot console=hvc0"

disk = ['file:/data/VMs/xenspark/System.img,xvda,w']

disk_other_config = []

veridian = 1

keymap = 'en-us'

maxmem = 1536

memory = 1024

name = 'xenspark'

on_crash = 'restart'

on_reboot = 'restart'

uuid = '47312d5c-b8b2-4aa1-aec0-9b1dd437f90b'

vcpus = 4

vif = ['bridge=home,mac=00:16:3E:49:4D:D2,type=netfront']

```

I have tried both Xen 4.1.0, and the unstable 9999 package

xl dmesg

```

nas configs # xl dmesg

 __  __            _  _    ____                     _        _     _

 \ \/ /___ _ __   | || |  |___ \    _   _ _ __  ___| |_ __ _| |__ | | ___

  \  // _ \ '_ \  | || |_   __) |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \

  /  \  __/ | | | |__   _| / __/|__| |_| | | | \__ \ || (_| | |_) | |  __/

 /_/\_\___|_| |_|    |_|(_)_____|   \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

(XEN) Xen version 4.2-unstable (@) (gcc version 4.4.5 (Gentoo 4.4.5 p1.2, pie-0.4.5) ) Sat Jul 23 10:15:56 PDT 2011

(XEN) Latest ChangeSet: Fri Jul 22 08:55:19 2011 +0100 23734:42edf1481c57

(XEN) Bootloader: GNU GRUB 0.97

(XEN) Command line:

(XEN) Video information:

(XEN)  VGA is text mode 80x25, font 8x16

(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds

(XEN)  EDID info not retrieved because no DDC retrieval method detected

(XEN) Disc information:

(XEN)  Found 5 MBR signatures

(XEN)  Found 5 EDD information structures

(XEN) Xen-e820 RAM map:

(XEN)  0000000000000000 - 000000000009f400 (usable)

(XEN)  000000000009f400 - 00000000000a0000 (reserved)

(XEN)  00000000000f0000 - 0000000000100000 (reserved)

(XEN)  0000000000100000 - 00000000b7ee0000 (usable)

(XEN)  00000000b7ee0000 - 00000000b7ee3000 (ACPI NVS)

(XEN)  00000000b7ee3000 - 00000000b7ef0000 (ACPI data)

(XEN)  00000000b7ef0000 - 00000000b7f00000 (reserved)

(XEN)  00000000b8000000 - 00000000c0000000 (reserved)

(XEN)  00000000f0000000 - 00000000f4000000 (reserved)

(XEN)  00000000fec00000 - 0000000100000000 (reserved)

(XEN)  0000000100000000 - 0000000140000000 (usable)

(XEN) ACPI: RSDP 000F7E60, 0014 (r0 Nvidia)

(XEN) ACPI: RSDT B7EE3000, 0038 (r1 Nvidia NVDAACPI 42302E31 NVDA        0)

(XEN) ACPI: FACP B7EE3080, 0074 (r1 Nvidia NVDAACPI 42302E31 NVDA        0)

(XEN) ACPI Warning (tbfadt-0444): Optional field "Pm2ControlBlock" has zero address or length: 0000000000000000/1 [20070126]

(XEN) ACPI: DSDT B7EE3100, 5B85 (r1 NVIDIA NVDAACPI     1000 MSFT  3000000)

(XEN) ACPI: FACS B7EE0000, 0040

(XEN) ACPI: SSDT B7EE8D80, 028A (r1 PTLTD  POWERNOW        1  LTP        1)

(XEN) ACPI: HPET B7EE9040, 0038 (r1 Nvidia NVDAACPI 42302E31 NVDA       98)

(XEN) ACPI: MCFG B7EE9080, 003C (r1 Nvidia NVDAACPI 42302E31 NVDA        0)

(XEN) ACPI: APIC B7EE8CC0, 0098 (r1 Nvidia NVDAACPI 42302E31 NVDA        0)

(XEN) System RAM: 3966MB (4061692kB)

(XEN) No NUMA configuration found

(XEN) Faking a node at 0000000000000000-0000000140000000

(XEN) Domain heap initialised

(XEN) found SMP MP-table at 000f3a30

(XEN) DMI 2.5 present.

(XEN) Using APIC driver default

(XEN) ACPI: PM-Timer IO Port: 0x1008

(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[1004,0], pm1x_evt[1000,0]

(XEN) ACPI:                  wakeup_vec[b7ee000c], vec_size[20]

(XEN) ACPI: Local APIC address 0xfee00000

(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)

(XEN) Processor #0 15:11 APIC version 16

(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)

(XEN) Processor #1 15:11 APIC version 16

(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)

(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)

(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])

(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])

(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])

(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])

(XEN) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])

(XEN) IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23

(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)

(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)

(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)

(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)

(XEN) ACPI: IRQ0 used by override.

(XEN) ACPI: IRQ2 used by override.

(XEN) ACPI: IRQ9 used by override.

(XEN) ACPI: IRQ14 used by override.

(XEN) ACPI: IRQ15 used by override.

(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs

(XEN) ACPI: HPET id: 0x10de8201 base: 0xfeff0000

(XEN) PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 63

(XEN) PCI: MCFG area at f0000000 reserved in E820

(XEN) Table is not found!

(XEN) Using ACPI (MADT) for SMP configuration information

(XEN) IRQ limits: 24 GSI, 376 MSI/MSI-X

(XEN) Using scheduler: SMP Credit Scheduler (credit)

(XEN) Detected 2612.084 MHz processor.

(XEN) Initing memory sharing.

(XEN) AMD K8 machine check reporting enabled

(XEN) AMD-Vi: IOMMU not found!

(XEN) I/O virtualisation disabled

(XEN) ENABLING IO-APIC IRQs

(XEN)  -> Using new ACK method

(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1

(XEN) Platform timer is 25.000MHz HPET

(XEN) Allocated console ring of 16 KiB.

(XEN) HVM: ASIDs enabled.

(XEN) SVM: Supported advanced features:

(XEN)  - Last Branch Record (LBR) Virtualisation

(XEN) HVM: SVM enabled

(XEN) AMD: Disabling C1 Clock Ramping Node #0

(XEN) Brought up 2 CPUs

(XEN) HPET: 3 timers (0 will be used for broadcast)

(XEN) ACPI sleep modes: S3

(XEN) MCA: Use hw thresholding to adjust polling frequency

(XEN) mcheck_poll: Machine check polling timer started.

(XEN) *** LOADING DOMAIN 0 ***

(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x835000

(XEN) elf_parse_binary: phdr: paddr=0x1a00000 memsz=0x9ba80

(XEN) elf_parse_binary: phdr: paddr=0x1a9c000 memsz=0xd60

(XEN) elf_parse_binary: phdr: paddr=0x1a9d000 memsz=0x12880

(XEN) elf_parse_binary: phdr: paddr=0x1ab0000 memsz=0x298000

(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1d48000

(XEN) elf_xen_parse_note: GUEST_OS = "linux"

(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"

(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"

(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000

(XEN) elf_xen_parse_note: ENTRY = 0xffffffff81ab0200

(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000

(XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"

(XEN) elf_xen_parse_note: PAE_MODE = "yes"

(XEN) elf_xen_parse_note: LOADER = "generic"

(XEN) elf_xen_parse_note: unknown xen elf note (0xd)

(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1

(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000

(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0

(XEN) elf_xen_addr_calc_check: addresses:

(XEN)     virt_base        = 0xffffffff80000000

(XEN)     elf_paddr_offset = 0x0

(XEN)     virt_offset      = 0xffffffff80000000

(XEN)     virt_kstart      = 0xffffffff81000000

(XEN)     virt_kend        = 0xffffffff81d48000

(XEN)     virt_entry       = 0xffffffff81ab0200

(XEN)     p2m_base         = 0xffffffffffffffff

(XEN)  Xen  kernel: 64-bit, lsb, compat32

(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1d48000

(XEN) PHYSICAL MEMORY ARRANGEMENT:

(XEN)  Dom0 alloc.:   0000000134000000->0000000138000000 (953726 pages to be allocated)

(XEN) VIRTUAL MEMORY ARRANGEMENT:

(XEN)  Loaded kernel: ffffffff81000000->ffffffff81d48000

(XEN)  Init. ramdisk: ffffffff81d48000->ffffffff81d48000

(XEN)  Phys-Mach map: ffffffff81d48000->ffffffff824aebf0

(XEN)  Start info:    ffffffff824af000->ffffffff824af4b4

(XEN)  Page tables:   ffffffff824b0000->ffffffff824c7000

(XEN)  Boot stack:    ffffffff824c7000->ffffffff824c8000

(XEN)  TOTAL:         ffffffff80000000->ffffffff82800000

(XEN)  ENTRY ADDRESS: ffffffff81ab0200

(XEN) Dom0 has maximum 2 VCPUs

(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff81835000

(XEN) elf_load_binary: phdr 1 at 0xffffffff81a00000 -> 0xffffffff81a9ba80

(XEN) elf_load_binary: phdr 2 at 0xffffffff81a9c000 -> 0xffffffff81a9cd60

(XEN) elf_load_binary: phdr 3 at 0xffffffff81a9d000 -> 0xffffffff81aaf880

(XEN) elf_load_binary: phdr 4 at 0xffffffff81ab0000 -> 0xffffffff81b2b000

(XEN) Scrubbing Free RAM: .done.

(XEN) Std. Loglevel: All

(XEN) Guest Loglevel: All

(XEN) Xen is relinquishing VGA console.

(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)

(XEN) Freed 224kB init memory.

(XEN) traps.c:2502:d0 Domain attempted WRMSR 00000000c0010004 from 0x00000182940c5083 to 0x000000000000abcd.

(XEN) PCI add device 00:00.0

(XEN) PCI add device 00:01.0

(XEN) PCI add device 00:01.1

(XEN) PCI add device 00:01.2

(XEN) PCI add device 00:02.0

(XEN) PCI add device 00:02.1

(XEN) PCI add device 00:04.0

(XEN) PCI add device 00:05.0

(XEN) PCI add device 00:06.0

(XEN) PCI add device 00:07.0

(XEN) PCI add device 00:08.0

(XEN) PCI add device 00:08.1

(XEN) PCI add device 00:09.0

(XEN) PCI add device 00:0b.0

(XEN) PCI add device 00:0c.0

(XEN) PCI add device 00:0d.0

(XEN) PCI add device 00:18.0

(XEN) PCI add device 00:18.1

(XEN) PCI add device 00:18.2

(XEN) PCI add device 00:18.3

(XEN) PCI add device 01:06.0

(XEN) PCI add device 02:00.0

```

Dom0 kernel xen configs

```

nas proc # zcat config.gz | grep -i xen

CONFIG_XEN=y

CONFIG_XEN_DOM0=y

CONFIG_XEN_PRIVILEGED_GUEST=y

CONFIG_XEN_PVHVM=y

CONFIG_XEN_MAX_DOMAIN_MEMORY=128

CONFIG_XEN_SAVE_RESTORE=y

# CONFIG_XEN_DEBUG_FS is not set

# CONFIG_XEN_DEBUG is not set

CONFIG_PCI_XEN=y

CONFIG_XEN_PCIDEV_FRONTEND=y

CONFIG_XEN_BLKDEV_FRONTEND=y

# CONFIG_XEN_BLKDEV_BACKEND is not set

CONFIG_XEN_NETDEV_FRONTEND=y

CONFIG_XEN_NETDEV_BACKEND=y

CONFIG_HVC_XEN=y

CONFIG_XEN_FBDEV_FRONTEND=y

# Xen driver support

CONFIG_XEN_BALLOON=y

CONFIG_XEN_SCRUB_PAGES=y

CONFIG_XEN_DEV_EVTCHN=y

CONFIG_XEN_BACKEND=y

CONFIG_XENFS=y

CONFIG_XEN_COMPAT_XENFS=y

CONFIG_XEN_SYS_HYPERVISOR=y

CONFIG_XEN_XENBUS_FRONTEND=y

CONFIG_XEN_GNTDEV=m

CONFIG_XEN_GRANT_DEV_ALLOC=m

CONFIG_XEN_PLATFORM_PCI=m

CONFIG_SWIOTLB_XEN=y

```

----------

## Moriah

I experienced the same problem when I upgraded from linux-2.6.31-gentoo-r6 to linux-2.6.37-gentoo-r4, which is what I am running now.  I was able to work around it by *MANUALLY* issuing /etc/init.d/vmware start, but if I do rc-update add vmware default, my vmware won't start; I must manually issue the command well after everything else has stabilized after a reboot.  Note that /etc/init.d/vmware start does not actually atart vmware; it only prepares the rest of the system to be able to run vmware.  I still need to start vmware itself manually, after I issue the above /etc/init.d/vmware start command.    :Confused: 

----------

## DingbatCA

Moriah, did you mean xend, not vmware?

Did some testing, and made sure "everything else has stabilized." still no luck.

```

nas ~ # uptime

 04:27:06 up 5 min,  1 user,  load average: 0.03, 0.16, 0.10

nas ~ # ps -ef | grep xen

root        22     2  0 04:21 ?        00:00:00 [xenwatch]

root        23     2  0 04:21 ?        00:00:00 [xenbus]

root      2011     1  0 04:21 ?        00:00:00 /usr/sbin/xenstored --pid-file=/                                                                                         var/run/xenstored.pid

root      2028     1  0 04:22 ?        00:00:00 /usr/sbin/xenconsoled --pid-file                                                                                         =/var/run/xenconsoled.pid

root      2044     1  0 04:22 ?        00:00:00 /usr/bin/python2.7 /usr/sbin/xe                                                                                          d start

root      2046  2044  1 04:22 ?        00:00:03 /usr/bin/python2.7 /usr/sbin/xe                                                                                          d start

root      2608  2600  0 04:27 pts/0    00:00:00 grep --colour=auto xen

nas ~ # ps -ef | grep xen

root        22     2  0 04:21 ?        00:00:00 [xenwatch]

root        23     2  0 04:21 ?        00:00:00 [xenbus]

root      2011     1  0 04:21 ?        00:00:00 /usr/sbin/xenstored --pid-file=/var/run/xenstored.pid

root      2028     1  0 04:22 ?        00:00:00 /usr/sbin/xenconsoled --pid-file=/var/run/xenconsoled.pid

root      2044     1  0 04:22 ?        00:00:00 /usr/bin/python2.7 /usr/sbin/xend start

root      2046  2044  1 04:22 ?        00:00:03 /usr/bin/python2.7 /usr/sbin/xend start

root      2610  2600  0 04:27 pts/0    00:00:00 grep --colour=auto xen

nas ~ # /etc/init.d/xend start

 * WARNING: xend has already been started

nas ~ # /etc/init.d/xend status

nas ~ # xm create /data/VMs/configs/xenspark.cfg

Using config file "/data/VMs/configs/xenspark.cfg".

Error: Device 51712 (vbd) could not be connected. Path closed or removed during hotplug add: backend/vbd/3/51712 state: 1

nas ~ # xm list

Name                                        ID   Mem VCPUs      State   Time(s)

Domain-0                                     0  3071     2     r-----     22.3

nas ~ # uptime

 04:28:38 up 7 min,  1 user,  load average: 0.23, 0.18, 0.11

nas ~ # /etc/init.d/xend restart

 * Stopping Xen control daemon ...                                                                                                                                 [ ok ]

 * Starting Xen control daemon ...                                                                                                                                 [ ok ]

nas ~ # xm create /data/VMs/configs/xenspark.cfg

Using config file "/data/VMs/configs/xenspark.cfg".

Error: Device 51712 (vbd) could not be connected. Path closed or removed during hotplug add: backend/vbd/4/51712 state: 1

```

----------

## Moriah

No, I actually did mean vmware, but the problem was so similar, I thought I'd mention it.

----------

