# Xen Hypervisor setup.

## xplosivelugnut

Hello everyone,

First, I'm very new to Linux. I started with Gentoo a few months ago and am capable of going from fresh install to plasma desktop fairly quickly. That said, I still have little idea what I'm doing.

My current goal is to setup a Gentoo Xen Domain0. The only DomU I'll end up having is a Windows 7 OS. Additionally, I'll have another system network booting off of the Dom0 machine but that's (probably) another thread.

Pulling bit's of information from several Xen guides (I have no idea how old or relevant these guides are), I think I have a working Dom0.

The first issues I ran into is that using xl hoses the console it was used in. For instance, if I do an " #xl list " it will print the top row of a table to the console and then hangs on the next line. It's process is unkillable. I've yet to get a DomU up and running so I'm wondering if this just happens becuase there aren't any to list?

The same thing occurs if I try to use " #xl create /path/to/example.cfg " It'll print out that it's parsing the config file and then hang on the next line.

Perhaps the biggest issue I'm having right now is that I don't know if I'm "in" Dom0. I have no idea how to tell the difference between the normal Gentoo environment and the Gentoo/Hypervisor environment. I used the "Gentoo with Xen Hypervisor" option in Grub. Was that it?

My reasoning for posting in the hardware section is that I'm trying to emulate hardware. Apologies if this belongs elsewhere.

----------

## The_Great_Sephiroth

I am in a similar boat as you, and nobody here seems to know anything about Xen. I have a nice server running XenServer 6.5 and I want to have a Gentoo guest, but I am not sure what kernel options and such I need, or if I should set any and simply use the kernel modules on the XS Tools disc. Totally lost here myself.

----------

## Atom2

 *xplosivelugnut wrote:*   

> My current goal is to setup a Gentoo Xen Domain0. The only DomU I'll end up having is a Windows 7 OS.

 That's feasable, but please be aware that unless your system supports VT-d for PCI passthrough and you have a supported GPU graphics device to be passed through to the Win7 domU, video output will be rather slow.

 *xplosivelugnut wrote:*   

> Pulling bit's of information from several Xen guides (I have no idea how old or relevant these guides are), I think I have a working Dom0.

 If you are running under XEN

```
# xl dmesg
```

will show you XEN's boot messages - which will also be shown on the console prior to booting the dom0. Also

```
# xl info
```

will provide further information about XEN like version, command line args, scheduler used, etc.

 *xplosivelugnut wrote:*   

> The first issues I ran into is that using xl hoses the console it was used in. For instance, if I do an " #xl list " it will print the top row of a table to the console and then hangs on the next line. It's process is unkillable. I've yet to get a DomU up and running so I'm wondering if this just happens becuase there aren't any to list?

 Even if there is no active domU, xl list should still display information about the Domain-0 (that's how dom0 is named in the xl list command). Also be aware that most (if not any) xl command requires root privileges to run and you could always add -v to get some more verbose output.

 *xplosivelugnut wrote:*   

> The same thing occurs if I try to use " #xl create /path/to/example.cfg " It'll print out that it's parsing the config file and then hang on the next line.

 Try the following command which will connect to the console of the to-be-created domU and displays it's boot messages:

```
# xl create path-to-config-file -c
```

 *xplosivelugnut wrote:*   

> Perhaps the biggest issue I'm having right now is that I don't know if I'm "in" Dom0. I have no idea how to tell the difference between the normal Gentoo environment and the Gentoo/Hypervisor environment. I used the "Gentoo with Xen Hypervisor" option in Grub. Was that it?

 You can use the command

```
# cat /proc/xen/capabilities

control_d
```

to see whether you are actually in the dom0. The control_d output denotes that you are in the controlling domain (i.e. dom0). For any domU that same command displays nothing.

 *The_Great_Sephiroth wrote:*   

> I am in a similar boat as you, and nobody here seems to know anything about Xen.

 Well, that's not entirely true - I have been using XEN for more than a year now and I am currently running a number of gentoo domUs (usually between 3 and 5 at anytime; all using a common r/o master image together with a dedicated r/w overlay per domU) and three FreeBSD domUs (1 pfSense firewall domU and two for Samba - although the latter two are currently under development) with the dom0 also being gentoo. All is absolutely stable and runs basically 24/7. The system is based on a Xeon processor with 32GB of ECC Ram and the motherboard supports Vt-d for PCI passthrough (mainly required for one of the FreeBSD domUs which receives a SATA controller in order to be able to provide ZFS storage, the pfSense firewall for it's WAN interface and the upcoming VoIP gentoo domU using telephony cards).

 *The_Great_Sephiroth wrote:*   

> I have a nice server running XenServer 6.5 and I want to have a Gentoo guest, but I am not sure what kernel options and such I need, or if I should set any and simply use the kernel modules on the XS Tools disc. Totally lost here myself.

 Please be aware that XEN != XenServer.

I have no experience whatsoever with XenServer but only the OpenSource XEN solution.Last edited by Atom2 on Sun Apr 12, 2015 10:46 am; edited 1 time in total

----------

## CaptainBlood

Hi, guys,

I agree gentoo's forum is somehow weak at supporting XEN, but very like the web itself, as mentionned.

Quite confusing the way Xen has grown up for the last 3 years (or so), not to mention integration level:

dom0 domu kernel settings

xend vs xl toolstacks dilemma

xen networking

virt-manager integration broken for xl toolstack.

grub2 & domUs

qemu as a side helper to xen (currently my difficult topic at noob level: how do/can they interact)

dom0 µcode 

MTRR

@Atom2: Your functionnal description is very much like mine, in progress work included. Architecture alike, too   :Surprised: 

So we probably faced similar difficulties at some point.

Good  :Idea:  your list of useful commands, for the almost neglected ones here,so far  :Embarassed: 

Thks 4 ur attention, interest & support.

----------

## The_Great_Sephiroth

I understand that, but my issues are essentially, what do I need in my kernel to run it under XS? I have asked on two separate occasions and never got a response. I use XS because we use it at my IT firm, I am very familiar with it, and it is way better on resources than Hyper-V, VMWare, or a few others we looked into. Also, I can pay for support for the host should I need it. Add to that clustering, failover, etc etc, and it is very appealing. All of which can be setup in a matter of minutes through the use of XenCenter.

----------

## Atom2

 *The_Great_Sephiroth wrote:*   

> I understand that, but my issues are essentially, what do I need in my kernel to run it under XS?

 Provided that XenServer is essentially comparable to XEN (which I do not know) then all depends on whether you want your gentoo system to run as an HVM guest (Hardware virtualized machine) or as a PV (paravirtualized) guest.

For the former (i.e. HVM) there is basically not much required as HVM emulates a virtual PC (including a BIOS) that gentoo (and any other system) can boot from. You define the (virtual) boot device (i.e. C or D) in the guests's config file.

In the guest system you may then also require a network driver if you want outside access through a NIC (you can choose from an Intel e1000 driver or a realtek rtl8139 driver, either of which you decide to use needs to be defined on the config file), a disk driver for the virtual disk interface (PIIX3 IDE) and a driver for s standard VGA card (Cirrus Logic or plain VGA emulated video; standard VGA is usally included in any kernel).

You can connect to the BIOS boot screen of the virtual PC (and afterwards to its virtual consoles) using a VNC viewer like TightVNC Viewer under Windows. Usually the guest system's NIC is connected through a bridge setup to the host system's (i.e. dom0's) NIC interface through which access to other system including the internet is being provided.

The disk space itself is usually being provided through the backend on the XEN dom0 and may be provided in a number of formats (i.e. file, LVM, etc). Standard OS drivers for the emulated devices should basically be available in almost all live CDs which may serve as the trampoline to install gentoo. HVM should work for any host system that runs under the x86/x64 architecture (x64 obviously only of if dom0 is also x64) and does not require any modification to the guest system. This means also standard windows OSes are able to run under HVM using their standard (included) drivers for the emulated devices.

PV on the other hand requires special paravirtualized drivers to provide disk and network IO which usually require patches / special drivers / special kernel parameters to be present for the guest OS. PV has many advantages over HVM and is not only quicker but also uses less ressources.

I suggest you first start familiarizing yourself with a HVM system to get an idea how all works together.

----------

## xplosivelugnut

Sorry, I never got notifications of replies. I see why, I'll be a bit more vigilent.

Thanks for said replies!   :Very Happy: 

The good news is that I've gotten it running. My problem was I hadn't set the Xen services to automatically start. If those services arn't running the xl command is broken and hangs in an unkillable state.

I've since had another issue however. I was trying to passthrough my GPU and one of the NICs. The NIC passed through flawlessly and my Windows 7 domU had network access. The GPU on the otherhand, while it was passed through, isn't able to restart/reset and can't be used. This apparently requires one of a special set of graphics cards. The Geforce 740 I have to work with isn't one of them.

I've had an idea I've yet to try, however.

The gentoo dom0 on this system will be headless. Only ssh and serial access. When the system boots, the GPU is used by the bios and then by grub followed by a little bit of kernel related stuff (?). Afterward some remaining lines of kernel stuff and a blinking cursor remains. When the Windows domU is started the text on the screen turns orange-ish and the shortly after the screen goes black indicating some activity.

Is there a way to prevent any use of the GPU beyond the bios and could that maybe allow the Windows domu to utilise it?

As of right now the card is visible in Window's device manager but has an "no resources available to be assigned to the device" error that is expected due to the GPU's inability to be reset. (using VNC to acctually view the Windows domU FYI)

Has anyone had experience with this hypothetical work arround?

----------

## keet

I had (and sort of still have) Xen working on my computer, but I never managed to make V.G.A. passthrough work.  V.G.A. passthrough in particular seems very tricky.  Windows Device Manager showed a code 43, and I read something about NVidia crippling their non-Quatro cards so that they would not work under a hypervisor.  I know that I'm being vague, but this was a month ago, and since then I've switched to K.V.M.  The way that I tried to make it work was by using my Intel C.P.U.'s built-in graphics so that Dom0 would not use the NVidia card at all.  I kept coming closer and closer to making it work, every little increment being a struggle.  I only really wanted it so that I would be able to play Windows games at full-speed without rebooting, and at this point, it's not worth the effort for me, compared to how little time I have for trying to make it work versus doing other things.

----------

## CaptainBlood

 *xplosivelugnut wrote:*   

> Sorry, I never got notifications of replies. I see why, I'll be a bit more vigilent.
> 
> Thanks for said replies!  
> 
> The good news is that I've gotten it running. My problem was I hadn't set the Xen services to automatically start. If those services arn't running the xl command is broken and hangs in an unkillable state.
> ...

 Hi, didn't read all your post, but AFAIK for VGA passthrough, plz make sure ur CPU has VT-d, & that it is enabled at bios level.

Once validated, everything required should be achieved at xen level.

Sorry but I have no VT-d hardware, so I can't help any further.

Thks 4 ur attention, interest & support

----------

## xplosivelugnut

 *keet wrote:*   

> I kept coming closer and closer to making it work, every little increment being a struggle.

 

Yeah, that's where I'm at right now.   :Laughing: 

 *CaptainBlood wrote:*   

> plz make sure ur CPU has VT-d, & that it is enabled at bios level.

 

Yup, it does and I do. The only thing stopping me now is the GPU.

----------

## xplosivelugnut

So, say I can't get the GPU passed through. Right now, I'm running a very basic Gentoo system (dom0). It's just a command line. I can switch between diffent "teminals" by holding Alt and using the left and right arrow keys, or hitting the F keys, etc... So is it possible to have this Windows 7 domU reside in one of the terminals. It would have to be wrapped in something. My first stab at it would be a GTK "window" taking up one of the teminals like a desktop/window manager does. Would this be possible without a desktop enviroment/window manager? I'd only have one window that would be "full screen".

This would require X and all that good stuff, but dealing with some overhead beats no GPU at all.

Pardon if my terminology is all wrong, still new to this. Hince all the quotes.

----------

## CaptainBlood

Hi,

So AFAIU u don't have graphics to M$, should it be without GPU passthrough.

In such case I would advice to unconfigure GPU passthrough & to concentrate on having a graphic session opened against Windows domU.

Plz note GPU passthrough shouldn't be required there, AFAIR

Thks 4 ur attention, interest & support

----------

## Atom2

 *xplosivelugnut wrote:*   

> So, say I can't get the GPU passed through. Right now, I'm running a very basic Gentoo system (dom0). It's just a command line. I can switch between diffent "teminals" by holding Alt and using the left and right arrow keys, or hitting the F keys, etc... So is it possible to have this Windows 7 domU reside in one of the terminals.

 As fas as I know that's not possible because the GPU is used by the linux driver (in text only mode) which prevents the windows domU from acquiring it (that would require PCI passthrough).

The virtual linux consoles that you are currently seeing (ALT-F1 .. ALT-Fn) are being provided by the linux dom0 and have nothing to do with any domU running on the system. Connections to a (text only) domU is either through ssh or by using a virtual (text) console from any of the virtual consoles by specifying the '-c' option to the xl create command.

What you, howver, can do is set up a HVM domU and install windows in that domU. This essentially emulates PC hardware in software including a BIOS with emulated devices like a Disk Controller, GPU, and a NIC. You would then require a VNC client which can either be run from the linux GUI (which you currently do not have) or you could connect from another (Windows) PC though e.g. TightVNC (I seem to remember that's Freeware) using the network. Please be aware that GPU performance using the emulated GPU is very slow (Windows 7's Aero Design doesn't work for instance), but apart from that you do have a fully working (virtual) Windows PC. For the virtual disk controller and the NIC there are (windows) drivers available that speed those up through paravirtualisation - but that can only be changed once the Windows system is up (and I think they are only available for Windows 32 bit as x64 requires signed drivers - although the more I think about it, I seem to remember that signed x64 drivers can actually be found somewhere).

The connection to the Windows domU would go through the LAN IP address of the dom0 (needs to be specified in the domU config file as by default the system only listens on 127.0.0.1) and port 5900 for the first domU (unless changed through the domU's config file). Access to the emulated virtual domU PC is available as early as the domU's BIOS is being displayed.

I hope that helps Atom2

----------

