# [solved] gentoo-sources-2.6.34-r2 and xen pci devices.

## Arrta

I decided to use the gentoo-sources-2.6.34-r2  kernel on a domU.

I got the DomU working, kernel loads, but lspci shows none of the PCI devices that I passed through.

I rechecked my kernel config and verified I enabled all the XEN devices. That is when I discovered that there is no CONFIG_XEN_PCIDEV_FRONTEND option anywhere in the gentoo-sources kernel.

According to the Xen docs, "Latest Linux kernels (2.6.27 and newer) are good for domU use." So I am not sure why it is missing.

Is the PCI frontend not in the linux base? or did it get renamed and I'm just not finding the ducumentation?

ArrtaLast edited by Arrta on Thu Aug 05, 2010 2:32 pm; edited 1 time in total

----------

## idella4

```

genny linux-2.6.34-gentoo-r2 # pwd

/usr/src/linux-2.6.34-gentoo-r2

genny linux-2.6.34-gentoo-r2 # grep CONFIG_XEN_PCIDEV_FRONTEND .config

genny linux-2.6.34-gentoo-r2 # cd ..

genny src # cd linux-2.6.34-xen

genny linux-2.6.34-xen # grep CONFIG_XEN_PCIDEV_FRONTEND .config

CONFIG_XEN_PCIDEV_FRONTEND=y

genny linux-2.6.34-xen # grep CONFIG_XEN_PCIDEV_FRONTEND /boot/config-2.6.34-gentoo-r2

genny linux-2.6.34-xen # grep CONFIG_XEN_PCIDEV_FRONTEND /boot/config-2.6.31-gentoo-r6

genny linux-2.6.34-xen # grep CONFIG_XEN /boot/config-2.6.31-gentoo-r6

genny linux-2.6.34-xen # grep CONFIG_XEN /boot/config-2.6.34-gentoo-r2

```

Forget the regular, you require the gentoo xen kernel

----------

## Arrta

I tried the xen-sources but they get stuck after 

```

kjournald starting.  Commit interval 5 seconds

EXT3-fs (xvda): mounted filesystem with writeback data mode

VFS: Mounted root (ext3 filesystem) readonly on device 202:0.

Freeing unused kernel memory: 296k freed

Write protecting the kernel read-only data: 7928k

kbd_mode used greatest stack depth: 5672 bytes left

init-early.sh used greatest stack depth: 4024 bytes left

```

and never boot into the system

----------

## idella4

You just need too adjust your xen kernel config.

It's not too hard.  Your kernel is mounting root, so it's very close.  You have most important things right already.

This has been trodden well before you.  Read closely this post and this other post and they should get you there.  Note the other links in those posts, they have the info.

If not, I can supply you with a config that will probably be a prefect fit, tomorrow

----------

## Arrta

If I can get the PCI devices to show to the gentoo-source-2.6.43-r2 that would work for me.

Otherwise I'm not sure what to do to fix the xen-source-2.6.34 unless it has to do with me not having pv-grub and a grub menu.list for the domU?

Edit: Seems not, pv-grub has something to do with booting off the local disk. I have a separate partition for the domU

Edit2: http://wiki.xensource.com/xenwiki/XenPCIpassthrough tells me that getting gentoo-sources to see the pci devices is not possible because "upstream kernel.org Linux kernel doesn't have Xen pcifront driver yet." So I have to use xen-sources at this time.

For the life of me I can't figure it out atm. My genkernel built gentoo-sources works, but the hand built xen-sources hangs at that step, genkernel won't work on xen.

Edit3: Not sure what changed, but genkernel is working for me with the xen-sources now. Previously it would bomb at the "Compiling 2.6.34-xen bzImage..." line. We will see how this turns out in a few mins once it finishes.

Edit4: Ok that failed miserably. Turns out I wasn't building off the config I loaded and genkernel picked its own without any options. I did however find out why my config is failing. There seems to be something in my config that turns off CONFIG_HAVE_KERNEL_BZIP2=y so I'll have to recreate the kernel config again and try to find it. That is unless you can get me that config you think will work.

Edit5: Last one, going to bed. Turns out enabling Xen-compatible disables the bzip option which causes genkernel to fail. I read in one of the posts to disable Xen-compatible for guests, but if I do that it also disables the PCI frontend. I thought setting Xen-compatible and "CONFIG_XEN_PRIVILEGED_GUEST is not set" would work, as when "CONFIG_XEN_PRIVILEGED_GUEST is not set" then "CONFIG_XEN_UNPRIVILEGED_GUEST=y", which in my opinion would be a domU.

There has to be something else I am missing, I'll take another look in the morning. Otherwise I think I will need a more obvious hint than those threads and the links inside of them. Maybe just getting me that config would work.

----------

## idella4

Arrta;

ok, let's get the environment clear.

 *Quote:*   

> 
> 
> I have a separate partition for the domU 
> 
> 

 

such as /dev/sda3.  My guess is, no, won't guess.  What is your host distro.  From all you've said, it could be debian.

Are you doing gentoo in gentoo or ??

Either way, you need a xen kernel to boot a gentoo guest.  It varies between distros.  Gentoo has obviously teased out xen content and concentrated on having it supplied in the gentoo prepared xen kernel.  emerge a vanilla source, different story.  You're not even using a gentoo-source, you're using a genkernel.  I spent two days recompiling a kernel before I got it to be adequate.  We'll get you there.

Next, post your gentoo.cfg.  That's fundamental.

PV-grub and pygrub work off grub.  Using these, you can in fact use your genkernel.  There is a condition.

They work off grub.  On booting, they firstly grab grub info from the guest.  The guest need have grub available to them.  That means the guest have grub installed to its mbr.  Having a partition for your guest places you at odds with pygrub.  Your grub for a system on a partition would normally install grub to the hard drive mbr.

What you need to equip your guest is to install grub to its own partition.

i.e. Instead of grub-install /dev/hda, you need grub-install /dev/hda3.

Then you can count on your genkernel.

It appears that you are using para-virt.  You haven't stipulated.  Now, the kernel. Let's get your xen kernel working.

In the xen kernel itself, if you stipulate

 *Quote:*   

> 
> 
>  │        [*] Support sparse irq numbering                                              │ │
> 
>   │ │        [*] Enable MPS table                                                          │ │
> ...

 

you get this in devices

 *Quote:*   

> 
> 
>  │        [*] Auxiliary Display support  --->                                           │ │
> 
>   │ │        <M> Userspace I/O drivers  --->                                               │ │
> ...

 

To enable your xen drivers as a host or guest,

 *Quote:*   

> 
> 
>   │ │        [*] Enable MPS table                                                          │ │
> 
>   │ │        [*] Enable Xen compatible kernel                                              │ │
> ...

 

yields

 *Quote:*   

> 
> 
> │        <M> Userspace I/O drivers  --->                                               │ │
> 
>   │ │            TI VLYNQ  --->                                                            │ │
> ...

 

namely

 *Quote:*   

> 
> 
>  │        [*] Privileged Guest (domain 0)                                               │ │
> 
>   │ │        <*> Backend driver support                                                    │ │
> ...

 

Leaving the first option  Xen compatible  left unchecked yields

 *Quote:*   

> 
> 
>  │        [ ] Enable Xen compatible kernel                                              │ │
> 
>   │ │        [*] Support for extended (non-PC) x86 platforms (NEW)                         │ │
> ...

 

viola; Paravirtualized guest support which is not so new.  check it, yields under device drivers

 *Quote:*   

> 
> 
>  │            TI VLYNQ  --->                                                            │ │
> 
>   │ │            Xen driver support  --->                                                  │ │
> ...

 

namely

 *Quote:*   

> 
> 
>  │        [*] Xen memory balloon driver (NEW)                                           │ │
> 
>   │ │        [*] Scrub memory before freeing it to Xen                                     │ │
> ...

 

Note: this enables your frontend xen drivers, most of them.

 *Quote:*   

> 
> 
>   Use the arrow keys to navigate this window or press the hotkey of │
> 
>              │  the item you wish to select followed by the <SPACE BAR>. Press    │
> ...

 

& you can now select Bzip2  This should get you there.  I hold an effective guest xen config in reserve.

```

idella@gentoo64 ~ $ ls /boot/config-2.6.31.6-xenU

/boot/config-2.6.31.6-xenU

/boot/config-2.6.32-xen-amd64-domU

```

----------

## Arrta

Gentoo is my host system, x86_64

My cpuinfo I got a core2quad:

```
processor       : 0-3

vendor_id       : GenuineIntel

cpu family      : 6

model           : 23

model name      : Intel(R) Core(TM)2 Quad CPU    Q8300  @ 2.50GHz

stepping        : 10

cpu MHz         : 2494.165

cache size      : 2048 KB

physical id     : 0

siblings        : 4

core id         : 3

cpu cores       : 4

apicid          : 3

initial apicid  : 3

fpu             : yes

fpu_exception   : yes

cpuid level     : 13

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority

bogomips        : 4986.73

clflush size    : 64

cache_alignment : 64

address sizes   : 36 bits physical, 48 bits virtual

power management:
```

Unmasked and installed on the host are:

```
sys-devel/dev86-0.16.17-r6 

sys-kernel/xen-sources-2.6.34

app-emulation/xen-tools-4.0.0

app-emulation/xen-4.0.0
```

My separate partition is actually /dev/md2. In my config I have this being loaded as xvda.

In my xen dom0 kernel I have:

In Processor type and features

```
[*] Support sparse irq numbering

[*] Enable Xen compatible kernel

[*] Single-depth WCHAN output

[*] Disable Bootmem code

    Processor family (Core 2/newer Xeon)  --->
```

In Bus options (PCI etc.)

```
[*] PCI support

[*]   Support mmconfig PCI config space access

[ ]   Xen PCI Frontend
```

In Device Drivers

```
< > Userspace I/O drivers  --->

    TI VLYNQ  --->

    XEN  --->

    Xen driver support  --->
```

In XEN

```
[*] Privileged Guest (domain 0)

<*> Backend driver support

<*>   Block-device backend driver

<*>   Block-device tap backend driver

<*>   Block-device tap backend driver 2

<*>   Network-device backend driver

(8)     Maximum simultaneous transmit requests (as a power of 2)

[ ]     Pipelined transmitter (DANGEROUS)

<*>     Network-device loopback driver

<*>   PCI-device backend driver

        PCI Backend Mode (Virtual PCI)  --->

[ ]     PCI Backend Debugging

< >   TPM-device backend driver

<*>   SCSI backend driver

<*>   USB backend driver

.......

[*] Disable serial port drivers

<*> Export Xen attributes in sysfs

(256) Number of guest devices

    Xen version compatibility (3.4.0 and later)  --->

[*] Place shared vCPU info in per-CPU storage
```

In Xen driver support

```
[*] Scrub memory before freeing it to Xen

<*> Xen /dev/xen/evtchn device
```

In my xen domU kernel I have:

In Processor type and features

```
[*] Support sparse irq numbering

[*] Enable Xen compatible kernel

[*] Single-depth WCHAN output

[*] Disable Bootmem code

    Processor family (Core 2/newer Xeon)  --->
```

In Bus options (PCI etc.)

```
[*] PCI support

-*-   Xen PCI Frontend

[ ]     Xen PCI Frontend Debugging
```

In Device Drivers

```
< > Userspace I/O drivers  --->

    TI VLYNQ  --->

    XEN  --->

    Xen driver support  --->
```

In XEN

```
[ ] Privileged Guest (domain 0)

< > Backend driver support

<*> Block-device frontend driver

<*> Network-device frontend driver

< >   Network-device frontend driver acceleration for Solarflare NICs

<*> SCSI frontend driver

<*> USB frontend driver

[*]   Taking the HCD statistics (for debug)

[ ]   HCD suspend/resume support (DO NOT USE)

<*> Framebuffer-device frontend driver

<*>   Keyboard-device frontend driver

[*] Disable serial port drivers

<*> Export Xen attributes in sysfs

(256) Number of guest devices

    Xen version compatibility (3.4.0 and later)  --->

[*] Place shared vCPU info in per-CPU storage
```

In Xen driver support

```
[*] Scrub memory before freeing it to Xen

<*> Xen /dev/xen/evtchn device
```

My grub/menu.lst for dom0

```
# For booting with disc 0 kernel

title Xen 4.0 / Linux 2.6.34-xen0 / (hd0,0)

root (hd0,0)

kernel /boot/xen.gz

module /boot/kernel-2.6.34-xen0 root=/dev/md1 pciback.permissive pciback.hide=(11:00.0)(11:01.0)(11:02.0)
```

This is so that I can pass through those 3 pci devices to one of the domU's

My dom0 runs fine and I have a Win 7 domU working off of /dev/md6.

I'm trying to install a gentoo domU on /dev/md2 this is where I am running into an issue.

I have the base system installed and if I use the gentoo-sources-2.6.34-r2 based kernel using para-virt the domU works. However, it can not see the PCI devices because "upstream kernel.org Linux kernel doesn't have Xen pcifront driver yet."

This is why I am trying to get the xen-sources on as a domU, because they have the "Xen PCI Frontend" option in them to accept the PCI devices passed through from the dom0.

Here is my config

```
# xen kernel

kernel = "/boot/kernel-2.6.34-xenU"

root   = "/dev/xvda ro raid=noautodetect"

# gentoo kernel

#kernel = "/boot/kernel-genkernel-x86_64-2.6.34-gentoo-r2"

#ramdisk = "/boot/initramfs-genkernel-x86_64-2.6.34-gentoo-r2"

#root   = "/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/xvda raid=noautodetect"

memory = 512

name   = "mythtv"

disk   = [ 'phy:md2,xvda,w' ]

vif    = [ 'mac=00:16:3E:2E:19:E9' ]

dhcp   = "dhcp"

pci    = ['11:00.0', '11:01.0', '11:02.0']

```

----------

## Arrta

Here are the options for the working gentoo-sources just to see what I have.

They are spread out all over the place so here is a grep of the .config instead.

```
linux-2.6.34-gentoo-r2 # grep XEN .config

CONFIG_XEN=y

CONFIG_XEN_MAX_DOMAIN_MEMORY=32

CONFIG_XEN_SAVE_RESTORE=y

# CONFIG_XEN_DEBUG_FS is not set

CONFIG_XEN_BLKDEV_FRONTEND=y

CONFIG_XEN_NETDEV_FRONTEND=y

CONFIG_XEN_KBDDEV_FRONTEND=y

CONFIG_HVC_XEN=y

CONFIG_XEN_FBDEV_FRONTEND=y

CONFIG_XEN_BALLOON=y

CONFIG_XEN_SCRUB_PAGES=y

CONFIG_XEN_DEV_EVTCHN=y

CONFIG_XENFS=y

CONFIG_XEN_COMPAT_XENFS=y

CONFIG_XEN_SYS_HYPERVISOR=y
```

Last edited by Arrta on Wed Aug 04, 2010 1:04 pm; edited 2 times in total

----------

## idella4

you are very close.  well you can now move on and try what I've pointed out.

Uncheck Xen compatable.

Check paravirt guest.

Enable its content.

correct your bzip2

re-compile

repost.

What about grub-install /dev/md...

I can get your environment now.  I think you have enough correct tips, you now  piece them together and execute,

and if you're really stuck, I'll give you the effective config which you just need to modify to your hardware

----------

## Arrta

 *idella4 wrote:*   

> you are very close.  well you can now move on and try what I've pointed out.
> 
> Uncheck Xen compatable.
> 
> Check paravirt guest.
> ...

 

Para-virt is not what I need. that does not allow the PCI frontend option in the PCI section

----------

## idella4

ok, let's ensure

 *Quote:*   

> 
> 
>  │        [*] Enable MPS table                                                          │ │
> 
>   │ │        [*] Enable Xen compatible kernel                                              │ │
> ...

 

 *Quote:*   

> 
> 
>  ┌──────────────────────────────────────────────────────────────────────────────────────┐ │
> 
>   │ │        [*] PCI support                                                               │ │
> ...

 

 *Quote:*   

> 
> 
>  ┌──────────────────────────────────────────────────────────────────────────────────────┐ │
> 
>   │ │        [*] Privileged Guest (domain 0)                                               │ │
> ...

 

 *Quote:*   

> 
> 
>            ┌───────────────────────── PCI Backend Mode ─────────────────────────┐
> 
>              │  Use the arrow keys to navigate this window or press the hotkey of │
> ...

 

As far as I can tell, your domU kernel wont work for the reason you specified, leaving you with the only option of gearing your host kernel as both host and guest.  From your post, you've not checked xen frontend under bus options yet.

Are you sure paravirt won't do it?

I'd suggest before you declare that finally, you have yet to install grub into the gentoo guest's own mbr,

i.e grub-install /dev/md2. So you have yet to declare that as successful.  Otherwise, you boot the hard drive itself which finds the grub menu.lst or grub.cnf.  That will be required for hvm.

There's only paravirt and hvm to choose.  

You haven't come to grips with pyrub.  It can boot the guest's resident kernel, in this case either regular or xen, and that's parvirt.

Oh, try switching your frontend xen drivers to modules, that's more the norm.

Anyway, have you tried your host kernel on the guest?

I'd like to see you get there, keep at it.

----------

## Arrta

I just compiled a xen-sources kernel enabling both frontend and backend. Previously I had compiled 2 separate kernels and only enabled the backend drivers on the dom0(host) kernel and the frontend drivers on the domU(guest) kernel. Am I misunderstanding the backend/frontend concept?

 *idella4 wrote:*   

> 
> 
> As far as I can tell, your domU kernel wont work for the reason you specified, leaving you with the only option of gearing your host kernel as both host and guest.  From your post, you've not checked xen frontend under bus options yet.
> 
> 

 

Thats because I thought frontend was only needed on the domU(guest), am I wrong?

 *idella4 wrote:*   

> 
> 
> Are you sure paravirt won't do it?
> 
> 

 

If you enable paravirt and disable xen compatable then the PCI frontend option does not even get displayed in Bus Options.

 *idella4 wrote:*   

> 
> 
> I'd suggest before you declare that finally, you have yet to install grub into the gentoo guest's own mbr,
> 
> i.e grub-install /dev/md2. So you have yet to declare that as successful.  Otherwise, you boot the hard drive itself which finds the grub menu.lst or grub.cnf.  That will be required for hvm.
> ...

 

I don't really understand why I need grub on the guest when the gentoo-sources paravirt works fine. Unless it is because the paravirt has some sort of code to not need grub.

And what do you mean by "boot the guest's resident kernel?" I thought I was already doing that.

I built the kernel in the chroot environ and then copied it out of the chroot to the host drive which is what I am using for the xen config.

 *idella4 wrote:*   

> 
> 
> Oh, try switching your frontend xen drivers to modules, that's more the norm.
> 
> 

 

will do.

 *idella4 wrote:*   

> 
> 
> Anyway, have you tried your host kernel on the guest?
> 
> 

 

not yet

 *idella4 wrote:*   

> 
> 
> I'd like to see you get there, keep at it.

 

Thanks, I'll keep trying. Need to wait 20 mins before I can restart with the new combo dom0/domU kernel.

----------

## idella4

Arrta

you have the concept quite right.  We're making headway. 

 *Quote:*   

> 
> 
> If you enable paravirt and disable xen compatable then the PCI frontend option does not even get displayed in Bus Options. 
> 
> 

 

Right, we're clear on that.  Let's get the basics in order.  You are booting a gentoo guest from a host which happens to be gentoo, could be any distro.  Your guest has no kernel installed.  It has a kernel, but let's day it's not installed as in actively installed with grub.

You can boot a guest by three means.  You are using the first option, using the kernel resident in the host, cited in the gentoo.cfg, to boot the guest.  From what you say, this is missing the required outcome of passing on some host hardware.  Now this I've touched on but not really firmly grasped myself.  Anyway, your domU kernel can't do it because of the above.  So, you seem to need a xen kernel with host capabilities that it can use to pass on the hardware to the guest.

Gentoo and xensource split the kernels into domU and dom0. Gentoo is just repeating xensource guides.

Look at the other two remaining distros that support xen, suse & debian, they have the one kernel that is packed with everything like a genkernel to match any host and any guest.  Grab the kernel config, look at it, it's huge.

Look at my gentoo's and in gentoo style it only has what's required.  Now you can take a suse / debian kernel config and cut it down to match, which is my reserve kernel config.

Pygrub and hvm install a guest complete with a kernel.  The usual option is the vbd, an image file or like you've got, a partition.  For them, grub is installed into an mbr.  The image file has an mbr and a partition. Well, grub can install anyway.  Now I take it you've created a gentoo on the partition manually.  You've not used virt-manager and virtinst, which is what I first did.  A partition is good since it's one single entity.  If you create an image file and install a guest vm into it, you can install a grub.  The easiest way is to not split it into /boot and / and /home, just like most installs, everything in /.  So, for paravirt, you can have a kernel in place or not in place.  pygrub and pv-grub need grub installed and a resident kernel.

So, use virt-manager / virt-install, it installs like a usual install, grub and kerneland all.  Booting it, the first step is to read the grub data from it's virtual mbr.  

Do you follow?  I'd like to see you boot it as you are, get your host kernel equipped with host & guest drivers, i.e. front & backend.  You can always acquire the xen docs to elaborate on all this.

----------

## Arrta

Ok, I missed the window at 11:00am EST. Had the contractor that is going to remodel our bathroom show up unexpectedly.

I have about hour before I have to switch the system back to a working state. Then at 6 EST I'm good for the rest of the night for testing.

At this point I'm not sure if you are telling me I need pygrub or pv-grub. I havent been able to find either of them having been installed with xen-tools even thou I reinstalled tools with the pygrub flag. I did read somewhere that pygrub is outdated and to use pv-grub, but all information I found says it is a part of xen-tools, which leads back to where is it, as xen-tools is not installing it.

Edit: Ok, well that bombed. Modified grub/menu.lst to boot from the new xen host, and they system is not comming back up. I'm working until 8 tonight so I won't be able to see what went wrong until at least 8:15ish when I get home. Depends on how many calls come in tonight.

The only things I changed were adding the frontend devices as modules and changed the PCI to Passthrough from Virtual PCI, I completely spaced on that setting.

----------

## idella4

Arrta

 *Quote:*   

> 
> 
> back to where is it, as xen-tools is not installing it.
> 
> 

 

Oh Oh Oh OH dear. 

```

idella@genny /mnt/suse/boot/grub $ ls /usr/lib/xen

bin  boot

idella@genny /mnt/suse/boot/grub $ ls /usr/lib/xen/bin/

imqebt    qemu-dm    stubdom-dm      xc_restore  xenconsole

lsevtchn  readnotes  stubdompath.sh  xc_save     xenctx

idella@genny /mnt/suse/boot/grub $ ls /usr/lib/xen/boot/

hvmloader  ioemu-stubdom.gz  pv-grub-x86_32.gz  pygrub

idella@genny /mnt/suse/boot/grub $ ls /mnt/gentoo64//usr/lib64/xen/boot/

hvmloader  ioemu-stubdom.gz  pv-grub-x86_32.gz  pv-grub-x86_64.gz  pygrub

idella@genny /mnt/suse/boot/grub $ ls /mnt/gentoo64//usr/lib64/xen/bin/

lsevtchn  readnotes   xc_restore  xenconsole

qemu-dm   stubdom-dm  xc_save     xenctx

```

Have a look and see.  However, you by rights don't need them.  They ought be there.

Now, pygrub is not out of fashion.  pv-grub might not come with gentoo's xen.

Style 1;  boot your guest with the kernel in the host /boot/kernel-xen-domU[or0]

Style 2: boot the guest with the guest's own kernel using pygrub or pv-grub, guest's regular [mostly] kernel in its /boot

Style 3:  boot the guest with the guest's own kernel using using hvm, as for pygrub.

Clear?

I'm not saying whether you need it or not.  These are the options.

You ought be able to do both.  Take your pick and get it straightened to that style.

Now you can't do both if pygrub's not there.

6 months ago I spent weeks working xen in gentoo, and after some event I can't be expected to remember, I discovered hvm and qemu-dm were MISSING. And here was I submitting xen type bugs.

xen packages have updated once or twice since then; seemingly having it fixed.

A week ago I got xen4.0.0 working in my genny.  At the moment it's broken.

I'll get it back.  It tells me that xen 4.0.0.should be good, but maybe not.

See I have both gentoo's xen and xensource xen.  For all I know gentoo still has a glaring bug in not getting the full content compiled.  If it misses it, I still get it from the xensource.  The versions are so close they'll be compatible.

Oh I just got it back, but it's still damaged.  See my post further down the list in this sub-forum, the one that' not getting any replies.

udev has broken xen's integrity.  gentoo64 is not updated and happily healthy on packages of 6 months  ago.

For now stick to your initial plan; get your kernel compiled as a dual host-guest, it's exactly what debian and suse do.

You need install grub only for option 2&3.

in between your job and your bathroom, we'll get you there. got it??

----------

## idella4

..

----------

## Arrta

So that was dunb, I enabled RAID0 and not RAID1 on my rebuilt kernel so it hung not being able to mount the md devices. 

Fixed that, but now we have taken a step backwards. These are on the guest.

Not sure why you thought this was solved..

```
[   12.926129] Root-NFS: No NFS server available, giving up.

[   12.926145] VFS: Unable to mount root fs via NFS, trying floppy.

[   12.926265] VFS: Cannot open root device "xvda" or unknown-block(2,0)

[   12.926273] Please append a correct "root=" boot option; here are the available partitions:

[   12.926283] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

[   12.926295] Pid: 1, comm: swapper Tainted: G        W  2.6.34-xen #5

[   12.926303] Call Trace:

[   12.926313]  [<ffffffff815bfa29>] panic+0x73/0x103

[   12.926323]  [<ffffffff81949114>] mount_block_root+0x1ce/0x1e5

[   12.926334]  [<ffffffff81002930>] ? trace_kmalloc+0x360/0xa30

[   12.926343]  [<ffffffff81949196>] mount_root+0x6b/0x8b

[   12.926349]  [<ffffffff81949326>] prepare_namespace+0x170/0x19d

[   12.926357]  [<ffffffff8194865c>] kernel_init+0x1d2/0x1e2

[   12.926364]  [<ffffffff810056c4>] kernel_thread_helper+0x4/0x10

[   12.926373]  [<ffffffff8194848a>] ? kernel_init+0x0/0x1e2

[   12.926382]  [<ffffffff810056c0>] ? kernel_thread_helper+0x0/0x10
```

I think this is because the frontend drivers are all modularized and I have no ramdisk.

..

Fixed that by using genkernel to make the ramdisk, now I'm stuck here.

```
[    0.423765] Initializing XFRM netlink socket

[    0.424058] NET: Registered protocol family 10

[    0.424752] ip6_tables: (C) 2000-2006 Netfilter Core Team

[    0.424793] IPv6 over IPv4 tunneling driver

[    0.425279] NET: Registered protocol family 17

[    0.425461] registered taskstats version 1

[    0.425589] PCI IO multiplexer device installed.

[    0.425603]   Magic number: 1:252:3141

[    0.425636] XENBUS: Device with no driver: device/vbd/51712

[    0.425641] XENBUS: Device with no driver: device/vif/0

[   12.938205] Freeing unused kernel memory: 404k freed

[   12.938382] Write protecting the kernel read-only data: 8560k
```

I'm gonna try building all the drivers in this time.

And now I'm back to the original problem with xen sticking at the following.

```
[    0.440211] kjournald starting.  Commit interval 5 seconds

[    0.440226] EXT3-fs (xvda): mounted filesystem with writeback data mode

[    0.440242] VFS: Mounted root (ext3 filesystem) readonly on device 202:0.

[    0.440384] Freeing unused kernel memory: 392k freed

[    0.440547] Write protecting the kernel read-only data: 8652k

[    1.522221] kbd_mode used greatest stack depth: 5672 bytes left

[    1.522628] init-early.sh used greatest stack depth: 4024 bytes left
```

I tried disabling the PCI frontend wondering if that is the cause, nope. I get stuck at the same point but a few hundred miliseconds sooner.

```
[    0.360063] kjournald starting.  Commit interval 5 seconds

[    0.360084] EXT3-fs (xvda): recovery complete

[    0.360590] EXT3-fs (xvda): mounted filesystem with writeback data mode

[    0.360612] VFS: Mounted root (ext3 filesystem) readonly on device 202:0.

[    0.360757] Freeing unused kernel memory: 392k freed

[    0.360917] Write protecting the kernel read-only data: 8644k

[    1.367494] kbd_mode used greatest stack depth: 5672 bytes left

[    1.367912] init-early.sh used greatest stack depth: 4024 bytes left
```

When the PCI frontend is active I do see the PCI Passthrough items get identified and loaded before the hangLast edited by Arrta on Thu Aug 05, 2010 3:49 am; edited 1 time in total

----------

## idella4

Arrta,

That message was a slip, i made it for another thread and entered it into yours instead.

```

root   = "/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/xvda raid=noautodetect"

```

This is a genkernel type of kernel args entry.  I can't tell if you've managed to actually boot it with genkernel or not.

A std para vm has a similar entry to a regular kernel.  I've never actually used genkernel.

root = "/dev/xvda1 ro console=tty0'

for your partition, yes, root=/dev/xvda

demo

```

idella@gentoo64 /mnt/genny/usr/src/linux-2.6.35-gentoo $ cat /etc/xen/gentoo.cfg

#                                                                               

# Configuration file for the Xen instance lenny01, created                      

# by xen-tools 4.1 on Sun May 16 01:10:35 2010.                                 

#  Hostname                                                                     

#                                                                               

name        = 'gentoo-2007'                                                     

#  Kernel + memory size

#

kernel      = '/mnt/squeeze/boot/kernel-2.6.31.13-xen-amd64-domU'

ramdisk     = '/mnt/squeeze/boot/initramfs.img-2.6.31.13-xen-domU'

memory      = '256'

#

#  Disk device(s).

#

root        = '/dev/xvda2 ro'

disk        = [

                  'file:/mnt/images/xen/images/gentoo.2007-0/gentoo.2007-0.img,xvda2,w',

                  'file:/mnt/images/xen/images/gentoo.2007-0/gentoo.swap,xvda1,w',

#                  'file:/mnt/images/xen/domains/Lenny.com/disk.img,xvdb,w',

                  'phy:sdb3,xvdc,w'

             ]

#

#  Physical volumes

#  Networking

dhcp        = 'dhcp'

vif         = [ 'mac=00:16:3E:59:C4:6E,bridge=eth0' ]

#vif         = [ ' ' ]

#

#  Behaviour

#

on_poweroff = 'destroy'

on_reboot   = 'restart'

on_crash    = 'restart'

vfb=['type=vnc,vncunused=1']

extra = '4 console=hvc0'

```

yes you need an initrd for your guest.

----------

## Arrta

I am able to boot the gentoo source built using genkernel in paravirt.

The genkernel randrive didnt help at all.

Building the frontend drivers into the kernel brought me back to the original issue.

Removing PCI Frontend had no effect on the "hang" other than making it happen slightly faster because the kernel did not detect and load the drivers for the 3 passed pci devices.

Going to test paravirt on the xen-sources with drivers built in and see what happens. I expect to be able to load up fine but without the passthrough items. Which will place me in the same situation as when I boot from the gentoo-sources paravirt kernel.

----------

## idella4

Just how are you passing the devices to the guest?  It seems to me they need be added in your

root='/dev/xvda ro console=tty0 pass-device-arg'

from man xm

 *Quote:*   

> 
> 
>    BLOCK DEVICES
> 
>        block-attach domain-id be-dev fe-dev mode [bedomain-id]
> ...

 

I don't know whether this would be included in xm create -c gentoo.cfg, or after it's booted and up.  If after, then you don't have a problem them not being accessed on boot.

----------

## Arrta

ok the paravirt xen-source failed at the same point as the non paravirt versions. "init-early.sh used greatest stack depth: 4024 bytes left"

This is how you pass PCI devices to the guest.

First find the addresses of the devices you want to pass

```
xen linux # lspci

11:00.0 Network controller: Techsan Electronics Co Ltd B2C2 FlexCopII DVB chip / Technisat SkyStar2 DVB card (rev 02)

11:01.0 Network controller: Techsan Electronics Co Ltd B2C2 FlexCopII DVB chip / Technisat SkyStar2 DVB card (rev 02)

11:02.0 Network controller: Techsan Electronics Co Ltd B2C2 FlexCopII DVB chip / Technisat SkyStar2 DVB card (rev 02)
```

On your dom0 add the following to the module line in your grub.lst and reboot

```
pciback.permissive pciback.hide=(11:00.0)(11:01.0)(11:02.0)
```

 This tells the dom0 not to load the PCI devices at thos locations.

In your domU config add this 

```
pci    = ['11:00.0', '11:01.0', '11:02.0']
```

 to allow the guest to see the PCI devices. From what I am able to tell in order to see these PCI devices the kernel has to be build with xen compatability in order to enable the PCI frontend in menuconfig.

There are more features like the ability to specify which slot to bind a device to 

```
pci = [ '11:02.0@7' ]
```

There is more information on this here http://wiki.xensource.com/xenwiki/VTdHowTo and here http://wiki.xensource.com/xenwiki/XenPCIpassthrough

Anyway, I'm going to bed now and I will try again tomorrow. If you could tell me where you are getting the "xensource" you refered to about xen-tools it might help me get working via pygrub. I tried the layman overlay called xen but that has xen, xen-tools, and xen-sources all with a 9999 revision. they download from a git tree and I cant get xen-tools to build.

```
gcc  -fno-strict-overflow -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement  -D__XEN_TOOLS__ -MMD -MF .subdirs-all.d  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE  -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement  -D__XEN_TOOLS__ -MMD -MF .subdir-all-blktap.d  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE  -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement  -D__XEN_TOOLS__ -MMD -MF .subdirs-all.d  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE  -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement  -D__XEN_TOOLS__ -MMD -MF .subdir-all-drivers.d  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE  -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -m64 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement  -D__XEN_TOOLS__ -MMD -MF .blktapctrl.d  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE  -Werror -Wno-unused -I../lib -I../../../tools/libxc -I../../../tools/include -I../../../tools/xenstore -I../../../tools/include -I ../../libaio/src -I ../../memshr -D_GNU_SOURCE -DUSE_GCRYPT -DMEMSHR -o blktapctrl blktapctrl.o blktapctrl_linux.o -Wl,-O1 -Wl,--as-needed                -L../../../tools/libxc -lxenctrl -L../../../tools/xenstore -lxenstore  ../../memshr/libmemshr.a -L../lib -lblktap -lrt -lm -lpthread

../../memshr/libmemshr.a(interface.o): In function `memshr_vbd_issue_ro_request':

/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools/memshr/interface.c:165: undefined reference to `xc_memshr_nominate_gref'

/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools/memshr/interface.c:179: undefined reference to `xc_memshr_share'

../../memshr/libmemshr.a(interface.o): In function `memshr_vbd_initialize':

/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools/memshr/interface.c:116: undefined reference to `xc_interface_open'

collect2: ld returned 1 exit status

make[4]: *** [blktapctrl] Error 1

make[4]: *** Waiting for unfinished jobs....

make[4]: Leaving directory `/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools/blktap/drivers'

make[3]: *** [subdir-all-drivers] Error 2

make[3]: Leaving directory `/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools/blktap'

make[2]: *** [subdirs-all] Error 2

make[2]: Leaving directory `/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools/blktap'

make[1]: *** [subdir-all-blktap] Error 2

make[1]: Leaving directory `/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools'

make: *** [subdirs-all] Error 2

make: Leaving directory `/var/tmp/portage/app-emulation/xen-tools-9999/work/xen-unstable.hg/tools'

```

----------

## idella4

Arrta

ok, I can follow that.  You're having a tough time with it.  That xen overlay appears to be like my xen-source, it looks the same.  Anyway, 

-your kernel building should be effective.  I suppose I'll give you the kernel config.

-Your system has no reason not to compile the xen tools.  I've got it working here, so probably fix that.  There's only a couple of points to have in place.  There is another approach to consider which is probably too long winded.

catch you tomorrow.

----------

## Arrta

haha, haha, haha,... *slams head on desk*

As you can tell I didn't goto bed. I took a look at the boot consol on the dom0, and it too got to the "init-early.sh used greatest stack depth:" but right after that it loads OpenRC. 

I remembered that my raid script complained about not being run on baselayout 2 so I emerged that and that I had to add OpenRC to my package.keywords on the dom0.

Since I copied all the configs over, it was i nthe guide I read, the same things got carried over. once I went removed openrc, and switched back to baselayout 1 for the domU and I got a different "hang" point, 

```
stty used greatest stack depth
```

 Technically the same one but it was more well documented.

I found someone who said they fixed it by changing the console being displayed to. Thats when I remembered I didn't have a console defined in my config.

once I copied your setting of 

```
extra = '4 console=hvc0'
```

 I got past the "hang" which wasn't actually a hang.

I now see in the console the system loading past the "hang" point and all my startup scripts ran. Once I fixed net.eth0 I was able to ssh into the domU with the xen-source running. Now I just have to readd the pci frontend into the kernel to see if the pci devces will show up with lspci. But, I'll do that tomorrow and confirm its solved at that point. Need sleep.

----------

## idella4

Arrta,

 well this really is becoming a protracted essay.  I didn't go to bed either, was following your progress.

good on ya mate the well expected breakthrough. 

once I copied your setting of 

That one important point is highlighted in the debian docs that you were never exposed to tackling it gentoo style.  I was missing that for a little while.  So that's good news, about time hey.  I helped overcome at least one important hurdle.

Your components of raid baselayout requirements are are different to mine, so I couldn't point that out, you needed to sort that yourself. Next is to tweak that kernel.  To clarify, does the bootup at this point complete to a login?  I've never cottoned on to using ssh which is heavily cited in using xen.

----------

## Arrta

No I do not get a login. xenconsole lets me watch the boot until "starting local" and then it just sits there. At that point I can ssh in to the system.

And WOOT!!!

```
xen linux # ssh 192.168.1.11

Password:

Last login: Thu Aug  5 02:04:26 EDT 2010 from 192.168.1.10 on pts/0

niflheim ~ # lspci

11:00.0 Network controller: Techsan Electronics Co Ltd B2C2 FlexCopII DVB chip / Technisat SkyStar2 DVB card (rev 02)

11:01.0 Network controller: Techsan Electronics Co Ltd B2C2 FlexCopII DVB chip / Technisat SkyStar2 DVB card (rev 02)

11:02.0 Network controller: Techsan Electronics Co Ltd B2C2 FlexCopII DVB chip / Technisat SkyStar2 DVB card (rev 02)
```

That is from within the domU. Now I can start on my planed setup.

----------

