# Xen ebuilds - discussion and directions

## trad511

The idea behind this topic is to begin or continue discussions around where to go with the Xen ebuilds in Gentoo.  Currently our ebuilds are for only the mainline Xen release, but discussions of supporting unstable, or Red Hat patches come up again and again in both bug reports and the forums - chiefly when a version bump is requested.  The bug that spawned this discussion is 179412 which is a version bump request from xen-3.0.4 to xen-3.1.0.   

As a brief aside, I am not a Gentoo dev.  I am simply a "power" user of Gentoo Xen on production servers since version 2 and on both production servers and everyday on my work laptop since version 3.  I will say that the Gentoo Xen devs have done a great job.  The Xen development line is historically far behind the mainline kernel and I know the devs get alot of pressure to support Xen for kernel versions that the main Xen has barely, if ever, touched.  Additionally, Xen requires a pretty hefty skill set to be a dev, and problems that arise due to bad builds can and have destroyed data.  As a final point, the Xen mailing lists leave something to be desired with respect to organization - it is very difficult to see what the direction of Xen is, who the players are or to find if your question has been answered in the past.  Please don't criticize or think I'm criticizing the the Gentoo Xen devs.  I simply see this support topic come up over and over and would like the community to have an open discussion.

A request was made in the current version bump bug 179412 to create an Xen ebuild which leverage the Fedora 2.6.20 patches or the (I hear) very beta 2.6.21 OpenSuSE patches.  I personally am beginning to get into hangup-ville since my ipw3945 drivers have changed significantly from my currently supported 2.6.16 xen kernel:  the ipw drivers/ebuild now are compatible with and require the ieee80211 stack in kernel 2.6.18 and later.  My current solution:  mask out ebuild updates.  I see this starting to happen now that gentoo main is getting further and further away from the supported Xen kernels.  As a consequence I'd love to see more recent kernels supported.

However, I see support of "third party" patching away from the Xen mainline as a risk, both from a data standpoint and from an ebuild maintenance standpoint. During the migration from Xen 3.0 to 3.0.2 (which then was eventually released as 3.0.4_p1) someone released ebuilds against the 2.6.18 kernel (sounds soooo old now doesn't it?) for the 2.6.16 xen patches.  This worked for several people, but one person used backend exported hardware (a SATA drive I believe) to a domU and really borked his data.  For a dev this isn't realistic for several reasons.  First, if these patches were designed for 2.6.16 and used on 2.6.18, any reports upstream to Xen will simply be discarded because it's not supported upstream.  Second, no dev wants to be responsible for maintaining an ebuild that seriously trashes data.

A similar thing may happen if we decide to support Fedora Xen patches.  The beauty of the open source community is that bugs are reported upwards towards the source.  But if we report FC-2.6.20 bugs to the Gentoo Xen devs where do they try to get them resolved?  Fedora won't support Gentoo's use of it's patches...

All this puts us in a difficult situation that has, as I see it, fractionalized the Gentoo Xen community.  The outcome of this same discussion during the 3.0 to 3.0.2/4 version bump request was the creation of several external, non-mainline, overlays.  While there's nothing inherently wrong with overlays, I think in this case that we now have several communities, each allied with the kernel version of Xen supported by the overlay ebuilds.  Doesn't really sound so good in the end to me since (that I know of) none of these communities share with each other.

What I'd like to suggest is putting mainline Xen (this means release + kernel version) in portage and the creation of an official Xen overlay for "buyer beware", unstable or third party (ie. FC) Xen ebuilds.  Obviously this requires some thought and planning with regard to support and communication, but that's what I wanted this forum topic to be about.  Comments?  Questions?  Suggestions?

----------

## smiler.se

 *trad511 wrote:*   

> What I'd like to suggest is putting mainline Xen (this means release + kernel version) in portage and the creation of an official Xen overlay for "buyer beware", unstable or third party (ie. FC) Xen ebuilds.  Obviously this requires some thought and planning with regard to support and communication, but that's what I wanted this forum topic to be about.  Comments?  Questions?  Suggestions?

 

I see a valid and good point in the problem with maintaining a "bleeding edge" official build of Xen. I think this overlay idea is a good solution if someone is up for really maintaining it.

For me, I need a recent kernel due to driver support/improvements, not because it's new. How does the Xen-people handle this issue? Alot has happend since 2.6.18, especially when it comes to ATA/SATA development.

----------

## roock

I am also using Gentoo + Xen in a production environment. And I would like to see xen with official patches in portage and one overlay for unstable xen. If i recall correctly, there were nearyl 3 or 4 overlays, when 3.0.4 went stable and 3.0.2 was in portage. So one official overlay with bleeding edge version would be very nice, in particular xen's official kernel is atm very much behind official kernel versions. And of course many thanks to the gentoo devs.. I would like to help more.. but spare time is rare...

----------

## tokka

 *trad511 wrote:*   

> 
> 
> What I'd like to suggest is putting mainline Xen (this means release + kernel version) in portage and the creation of an official Xen overlay for "buyer beware", unstable or third party (ie. FC) Xen ebuilds.

 

That sounds like the way to go to me.

I've got a production server that has been running Xen 2 for ages, it has has no problems and so I've left it alone. Two weeks ago I decided to build a new Xen server, what was in Portage was ancient, gentoo-wiki.com pointed me towards two overlays with 3.03 and 3.04. I tried 3.04 and it seems to be ok bar networking issues covered in this forum -  then to my surprise the 3.04 ebuild seemed to get stuffed into Portage without the known issues being addressed...

So currently a Gentoo newcomer to Xen using the ebuilds in Portage will end up with a real headache trying to get networking working - whereas other major distributions have pretty slick Xen implementations - before the 3.1 release I found myself seriously considering using a CentOS domain 0 with Gentoo VM's :Sad: 

I think a start would be to have working ebuilds in Portage, and then yes an official overlay, there are already people posting in this forum having issues with the mechanics of installing an ebuild from that bug thread.

----------

## CpuID

Im all for official releases in portage, and bleeding edge/experimental stuff in an overlay. I just gave the OpenSUSE patches a test against current gentoo-sources-2.6.21*, and its rather nasty, lets just say that much  :Smile:  The xen3-patch-2.6.21 has no chance of applying in its current state, not to mention the mach-xen asm dirs that dont exist for some archs, im considering giving the redhat patches a try next against 2.6.20 or something... Let me know if anyone starts something to collaborate on progress in different areas etc, maybe a xen overlay on overlays.gentoo.org or something? marineam might like to push his progress over that way maybe... either that or make more use of the VPS overlay I guess...

----------

## sdrik

I've just published my overlay containing my 2.6.20 Xen kernel, based on Fedora patches :

http://cedric.gabriello.fr/gentoo/layman.xml

http://cedric.gabriello.fr/gentoo/sdrik-xen.tar.bz2

Only DomU/i386 has been tested for now as my Dom0 is an Ubuntu system, but I use nearly the same patchset for my Ubuntu packages (http://cedric.gabriello.fr/ubuntu)

----------

## tomekki

Hi,

I just gave it a try and compiled a dom0 kernel on my x86_64 architecture. 

It does not work:

```

linux-2.6.20.12-xen # make

  

  ...

  CC      drivers/xen/util.o

  LD      drivers/xen/privcmd/built-in.o

  CC      drivers/xen/xenbus/xenbus_xs.o

  CC      drivers/xen/xenbus/xenbus_probe.o

  CC      drivers/xen/xenbus/xenbus_backend_client.o

  CC      drivers/xen/xenbus/xenbus_probe_backend.o

  CC      drivers/xen/xenbus/xenbus_dev.o

  LD      drivers/xen/xenbus/xenbus_be.o

  LD      drivers/xen/xenbus/built-in.o

  LD      drivers/xen/built-in.o

  LD      drivers/built-in.o

  GEN     .version

  CHK     include/linux/compile.h

  UPD     include/linux/compile.h

  CC      init/version.o

  LD      init/built-in.o

  LD      .tmp_vmlinux1

arch/x86_64/kernel/built-in.o: In function `intel_bugs':

early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'

make: *** [.tmp_vmlinux1] Error 1

```

As told, it was not tested completely ...

----------

## CpuID

hey sdrik, I just used the linux-2.6-xen.patch out of the 2.6.20 2925 srpm for fedora core 7, no other patches. patched against gentoo-sources 2.6.20-r8 and it built and booted first time with x86_64  :Smile:  anything else your patchset has that I dont know about? only thing left to sort out is my missing phys_to_machine symbol for nvidia drivers, which shouldnt be too hard

----------

## zoltak

 *CpuID wrote:*   

> hey sdrik, I just used the linux-2.6-xen.patch out of the 2.6.20 2925 srpm for fedora core 7, no other patches. patched against gentoo-sources 2.6.20-r8 and it built and booted first time with x86_64  anything else your patchset has that I dont know about? only thing left to sort out is my missing phys_to_machine symbol for nvidia drivers, which shouldnt be too hard

 

You will need the SMP fix if you want your virtual machines to access more then 1 VCPU's. Thats all I found so far.

----------

## sdrik

 *CpuID wrote:*   

> hey sdrik, I just used the linux-2.6-xen.patch out of the 2.6.20 2925 srpm for fedora core 7, no other patches. patched against gentoo-sources 2.6.20-r8 and it built and booted first time with x86_64  anything else your patchset has that I dont know about? only thing left to sort out is my missing phys_to_machine symbol for nvidia drivers, which shouldnt be too hard

 

patch 01 is the core xen patch

patches 10-13 are things in fedora's patchset which touches xen bits and looked interesting

patches 20-25 contain a fix for a nasty problem which triggers when using dhcp beetween DomU and Dom0

patches 50-51 are random fixes necessary to build (but may be needed only with some specific configs)

patch 90 is needed to build with binutils-2.16.1 (the problem reported in bug #179412 comment #22)

I've only tried with vanillia sources, so maybe some patches are not needed anymore with gentoo-sources

----------

## sdrik

 *tomekki wrote:*   

> Hi,
> 
> I just gave it a try and compiled a dom0 kernel on my x86_64 architecture. 
> 
> It does not work:
> ...

 

I cannot reproduce your problem here, it must be specific to your .config

Can you send me your .config privately ? I don't think this topic is the good place to further debug my overlay (where can this take place ? I'm new to Gentoo hacking)

----------

## zoltak

Is anyone having problems using images via 

```
tap:aio:/

```

I get an error:

```
XENBUS: Timeout connecting to device: device/vbd/51712 (state 3)

XENBUS: Timeout connecting to device: device/vbd/51715 (state 3)

XENBUS: Timeout connecting to device: device/vbd/51714 (state 3)
```

File method works fine

```
file:/
```

The images boot on a centos dom0 running 2.6.18

----------

## CpuID

theres an updated overlay at http://www.nightsys.net/xen-3.1.0-2.6.20.12-overlay.tar.bz2 , which includes sdriks patches (thanks sdrik, they all look pretty good  :Smile: ) for the kernel, mixed with gentoo-sources genpatches 10 (had to exclude the 2.6.20.y patches from genpatches to avoid duplication but otherwise they work fine), also included is xen-3.1 and xen-tools-3.1 (thanks gentoo@soit.de), and the nvidia-drivers-9755-r2 patched to work with xen (thanks Guy Baconniere). hope everyone likes, im typing this from a dom0 using the above, with the nvidia drivers working fine  :Smile:  I attempted to boot a winxp pro hvm last night, it got into setup fine, did the first restart, but hung right on the end of installing devices..., but i think thats most likely a driver issue with xp setup than xen if anything (it hung halfway through installing devices at first, turned off pae and it got to almost the end)...

----------

## zoltak

 *zoltak wrote:*   

> Is anyone having problems using images via 
> 
> ```
> tap:aio:/
> 
> ...

 

I don't need kernel 2.6.20 on Dom0 so I installed 2.6.18 and the problem above goes away. I still use the patched 2.6.20 for domU's  :Smile: 

Has anyone else overcome this problem running 2.6.20 on Dom0 with tap:aio devices?

----------

## sdrik

 *zoltak wrote:*   

>  *zoltak wrote:*   Is anyone having problems using images via 
> 
> ```
> tap:aio:/
> 
> ...

 

Kernels built from upstream xen tarballs (this his the case for the 2.6.18 contained in my overlay) contain a patch (blktap-aio-16_03_06.patch) which has been vetoed upstream. Fedora decided not to include it so they implented another patch against xen tools as a workaround (xen-blktap-no-aio-epoll.patch)

(http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00331.html)

I will work on integrating both xen-blktap-no-aio-epoll.patch and blktap-aio-16_03_06.patch in my xen-sources and (upcoming) xen ebuilds and let the user select between the two methods by a USE flag.

----------

## zoltak

Could someone send me a working .config for 2.6.20 privatley or post a link. 

Thanks

----------

## tomekki

Hi,

I still can't compile a (domU) Kernel. I tried sdrik's and CpuID's patched version without success.

Fortunately, I was able to isolate the kernel option responsible for my previous error. Which was:

```

arch/x86_64/kernel/built-in.o: In function `intel_bugs':

early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'

make: *** [.tmp_vmlinux1] Error 1 

```

The option which lets the compilation fail is:

```

Bus options (PCI etc.)  ---> 

[*] PCI support

```

At least I can make my very first steps with Xen  :Smile: 

However, I would like to have PCI available in my future domU machine.

Any Idea what might went wrong with the the PCI support?

----------

## zoltak

Is it possible to specify 2 ip's on eth0 in dom0. When I do this it fails to bring eth0 up? 1 works fine?

Any suggestions?

----------

## Mescalito

Hey,

I made an ebuild and some layman stuff for it.

emerge -uv layman

layman -kfo http://thestonertree.com/layman.txt -a mescalito

emerge -av xen xen-tools xen-sources

----------

## CpuID

 *tomekki wrote:*   

> Hi,
> 
> I still can't compile a (domU) Kernel. I tried sdrik's and CpuID's patched version without success.
> 
> Fortunately, I was able to isolate the kernel option responsible for my previous error. Which was:
> ...

 

To be honest, I just hashed this quirk out, as I dont have the hardware it relates to (according to the comments in the src)  :Smile:  But yea, from what I can tell the quirk exists in i386 but wasnt replicated to amd64, but im reluctant to do so as im not experienced enough to know if its safe/what the implications are in the case you do have that hardware  :Razz: 

----------

## Mescalito

 *CpuID wrote:*   

>  *tomekki wrote:*   Hi,
> 
> I still can't compile a (domU) Kernel. I tried sdrik's and CpuID's patched version without success.
> 
> Fortunately, I was able to isolate the kernel option responsible for my previous error. Which was:
> ...

 

The problem with disabling [*] PCI support is you disable support for other kernel options, and to be honest it's more likely that one of these dependant options is to blame than PCI support itself.

I put the 2.6.20 kernel patches in my repository and am running WinXP inside it now with nvidia drivers on linux. Everything seems to work amazingly.

----------

## tomekki

Hi Mescalito,

  i just tried your xen overlay and this time the compilation of a kernel 2.6.18 went smoothly.

Thanks, now I can start to use pci devices in domU.  :Smile: 

You mentioned that you checked in kernel 2.6.20 patches in your repository. 

I could find those under 'svn://thestonertree.com/mescalito/sys-kernel/xen-sources'. Could you give me a hint where to find the patch?

I agree, I would be also afraid that missing PCI support would bring some unexpected side effects. 

Btw. CpuID, I hope that Santa is going to give you a nice 64 bit machine  :Wink: 

Greetings

----------

## CpuID

tomekki, I already have 3 amd64 boxen  :Razz:  a c2d t7200 mediapc (1gb ram, 32bit gentoo only atm though), c2d e6600 home desktop (with 4gb ram, 64bit gentoo), and an opteron 148 work desktop (with 4gb ram, 64bit gentoo) hehe.

----------

## Mescalito

 *tomekki wrote:*   

> Hi Mescalito,
> 
>   i just tried your xen overlay and this time the compilation of a kernel 2.6.18 went smoothly.
> 
> Thanks, now I can start to use pci devices in domU. :)
> ...

 

Do a `svn update' or use `layman -s mescalito'

Cheers, James

----------

## tomekki

At the risk of doing wrong ...

I just found an ebuild for kernel 2.6.18:

```

svn co svn://thestonertree.com/mescalito

[...]

A    mescalito/app-emulation/xen/xen-3.1.0.ebuild

A    mescalito/sys-kernel

A    mescalito/sys-kernel/xen-sources

A    mescalito/sys-kernel/xen-sources/metadata.xml

A    mescalito/sys-kernel/xen-sources/files

A    mescalito/sys-kernel/xen-sources/files/digest-xen-sources-2.6.18

A    mescalito/sys-kernel/xen-sources/files/patch-2.6.18_to_xen-3.1.0.bz2

A    mescalito/sys-kernel/xen-sources/Manifest

A    mescalito/sys-kernel/xen-sources/ChangeLog

A    mescalito/sys-kernel/xen-sources/xen-sources-2.6.18.ebuild

Checked out revision 5.

```

Might it be that your patches for kernel 2.6.20 are not accessible for anonymous svn users?

But, no stress ... I live already quite well with the 2.6.18 kernel

Greetings, Tomek

----------

## Mescalito

 *tomekki wrote:*   

> At the risk of doing wrong ...
> 
> I just found an ebuild for kernel 2.6.18:
> 
> ```
> ...

 

Ah weird you're right I missed them. Strange because at some point I definitely used it to install my xen system.

James

----------

## solussd

Is there a working, relatively stable ebuild for 2.6.20-xen?

I have a new quad core xeon system (x86_64) with 3 sata drives set up in software raid. needless to say, the current xen-sources (2.6.16.49) do not support my hardware very well (cant boot xen kernel!!)

would 2.6.18-xen work with sata (ahci) ?

----------

## tomekki

Hi solussd,

I am using 2.6.18-xen kernel with a combination of Raid 0 and 1 in Dom0 and it works fine for me.

I have an Intel Core Duo (x86_64) machine. 

What is not working for me, is to build a DomU 2.6.20-xen kernel on my x86_64 architecture.

Then I am always receiving the error:

```

arch/x86_64/kernel/built-in.o: In function `intel_bugs':

early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'

make: *** [.tmp_vmlinux1] Error 1 

```

If you are able to build a DomU 2.6.20-xen kernel on your similar architecture let me know  :Wink: 

Greetings, Tomek

----------

## solussd

tomekki- thanks for the reply

I (finally) got my sata software raid working using 2.6.16. I am able to build a domU just fine-- but only if I take out the serial 8250 stuff. I haven't tried a 2.6.20 kernel yet but I will probably be trying either that or 2.6.18 today. did you have any issues with 2.6.18?

----------

## tomekki

Hi solussd,

 just my on-board intel video card (Intel P965+ ICH8 chipset) is not working under X11, but this seems to be a general problem with the default 2.6.18 kernel independent of Xen.

Otherwise I did not run into problems so far.

Greetings, Tomekki

----------

## sithmein

 *tomekki wrote:*   

> 
> 
> Then I am always receiving the error:
> 
> ```
> ...

 

Same problem here. Has anyone solved this without disabling PCI?

----------

## shakedown

 *sithmein wrote:*   

> 
> 
> Same problem here. Has anyone solved this without disabling PCI?

 

if you have an amd you could comment it out. 

worked for me.

arch/x86_64/kernel/early-quirks.c

```

/*

static void intel_bugs(void)

{

        u16 device = read_pci_config_16(0, 0, 0, PCI_DEVICE_ID);

#ifdef CONFIG_SMP

        if (device == PCI_DEVICE_ID_INTEL_E7320_MCH ||

            device == PCI_DEVICE_ID_INTEL_E7520_MCH ||

            device == PCI_DEVICE_ID_INTEL_E7525_MCH)

                quirk_intel_irqbalance();

#endif

}

*/

struct chipset {

        u16 vendor;

        void (*f)(void);

};

static struct chipset early_qrk[] = {

        { PCI_VENDOR_ID_NVIDIA, nvidia_bugs },

        { PCI_VENDOR_ID_VIA, via_bugs },

        { PCI_VENDOR_ID_ATI, ati_bugs },

/*      { PCI_VENDOR_ID_INTEL, intel_bugs},*/

        {}

};

```

----------

## newtonian

 *tomekki wrote:*   

> Hi solussd,
> 
> What is not working for me, is to build a DomU 2.6.20-xen kernel on my x86_64 architecture.
> 
> Then I am always receiving the error:
> ...

 

Try this in your domU config: 

Processor type and features  --->

and uncheck: Symmetric multi-processing support

Cheers,

----------

