# [SOLVED] Clone gentoo-sources Core2 to Linode (Xen)

## jeffk

I migrated a physical ~amd64 Core2 gentoo linux instance to Linode VPS, which is Xen-based. Works great under a chroot, and I'd like to enable a kernel for a true boot.

Do I need to switch from gentoo-sources to xen-sources, or will genkernel reconfiguration (menuconfig) be sufficient?

The migration docs only specify pvops (paravirt ops) as a necessary kernel configuration. I presume this means PARAVIRT_GUEST and related options.

My current physical machine config is as follows:

```
# grep -i virt kernel-config-x86_64-2.6.34-gentoo-r1 

# CONFIG_PARAVIRT_GUEST is not set

CONFIG_VIRT_TO_BUS=y

# CONFIG_NET_9P_VIRTIO is not set

# CONFIG_VIRTIO_BLK is not set

# CONFIG_VIRTIO_NET is not set

# CONFIG_VIRTIO_CONSOLE is not set

# CONFIG_HW_RANDOM_VIRTIO is not set

# CONFIG_FB_VIRTUAL is not set

# CONFIG_SND_VIRTUOSO is not set

# CONFIG_DEBUG_VIRTUAL is not set

CONFIG_VIRTUALIZATION=y

CONFIG_VIRTIO=m

CONFIG_VIRTIO_RING=m

CONFIG_VIRTIO_PCI=m

CONFIG_VIRTIO_BALLOON=m
```

http://library.linode.com/advanced/migrate-server-to-linode

http://library.linode.com/advanced/pv-grub-howto

http://library.linode.com/advanced/custom-distro-howto

Thanks.Last edited by jeffk on Fri Jul 16, 2010 4:09 pm; edited 1 time in total

----------

## kimmie

My understanding of Xen is that you do not have direct control over your own kernel, the supervisor loads the hosted kernel from it's own filesystem space. If the hosting provider is kind, they will compile modules for you and provide a means of getting them into your virtual filesystem so you can load them. I had an experience with a provider that refused to do this, so I ditched them. Of course, I could be wrong here and I'm not ensurely what the situation wiht linode is, although I looked at your links below and the final linode steps are asking you to choose one of their kernels, which is appropriate to you distro.

Sorry if I'm not helpful, but in that case maybe I can learn something from you?

----------

## idella4

Could you clarify what it is you're trying to do?  A true boot, ???

I know xen quite well but this description I can't follow.  From http://library.linode.com/advanced/migrate-server-to-linode,  you migrate an established vm into Finnix as a para-virt guest.

Finnix is a host or a guest? It's listing /dev/xvda b c,  These are para-virt xen devices char files.

Is finnix listing these device choices for the guest it's working on, or is is a guest vm itself?????

Overall, what host environment are you going to work from?

I've acquired installed and run a xen live cdm so I'm familiar with xen, but this finnix is new. 

This migration process ends with booting up the migrated virtual server guest.  That is done by a host linux o/s in a xen environment.

What is the true boot?  You have a vm of gentoo which has been 'run' by a xen based linode structure of some sort.

Do you want to create a xen host and boot it up on your home pc?

I just won't add although I know much about xen when 	I can't fathom what you're trying for.

If you make that clear, I should be able to offer something.

----------

## jeffk

 *idella4 wrote:*   

> Could you clarify what it is you're trying to do?  A true boot, ???
> 
> I know xen quite well but this description I can't follow.  From http://library.linode.com/advanced/migrate-server-to-linode,  you migrate an established vm into Finnix as a para-virt guest.

 

Thanks for the reply. Yes, the terminology eludes me as yet. I'm familiar with kvm virtualization, this is my first encounter with xen. 

In simplest terms, I want to make my linode xen guest running Gentoo as similar to the physical machine I'm migrating from as possible. This is a one time migration, currently I have the whole fileystem copied to the linode. Since I have some things  that require specific kernel config, I would like to configure and boot from my own genkernel using PV-GRUB, rather than use a linode-provided kernel.

This is a new link I just found this morning: http://www.linode.com/wiki/index.php/PV-GRUB#Building_Your_Own_Kernel

 *idella4 wrote:*   

> Finnix is a host or a guest? It's listing /dev/xvda b c,  These are para-virt xen devices char files.
> 
> Is finnix listing these device choices for the guest it's working on, or is is a guest vm itself?????
> 
> Overall, what host environment are you going to work from?

 

I haven't used finnix, but I gather its a linode-custom system rescue cd type distribution.

I'm not using finnix, although it may come in handy to diagnose boot issues. I'm using a gentoo ~amd64, which I copied from my phyicial machine to the linode disk.

I am keeping the filesystem-cloned Gentoo xen guest (which has not booted yet) updated under chroot from an Ubuntu 10.04 instance in the linode (i.e. xen guest).

----------

## idella4

jeffk, 

ok, we will get there.   I've practiced with xen extensively and I have working versions of PV-GRUB, but just spell this out a little more.

 *Quote:*   

> 
> 
> its a linode-custom system rescue cd type distribution. 
> 
> 

 

finnix is a customized linux that runs as a host.  It has linode tools in it that appear in the link.

It is a xen host that manages xen guests.  It therefore has a kernel that is running it as a host.

Is this a fair summary?

 *Quote:*   

> 
> 
> I'm using a gentoo ~amd64, which I copied from my phyicial machine to the linode disk. 
> 
> 

 

Just like me.  You're running a regular gentoo system from a regular kernel?  So if you enter

uname -a,

you get something like

```

gentoo64 linux-2.6.34-gentoo-r1 # uname -a

Linux gentoo64 2.6.34-gentoo-r1-amd64 #2 SMP Wed Jul 14 09:42:51 Local time zone must be set--see zic  x86_64 Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz GenuineIntel GNU/Linux

```

Is this correct?

 *Quote:*   

> 
> 
> I want to make my linode xen guest running Gentoo as similar to the physical machine I'm migrating from as possible
> 
> 

 

physical machine, as in the hardware components, or machine as in a linux installed system machine???

By machine, a computer, or a system as in virtual machine???

 *Quote:*   

> 
> 
> Which distributions do you offer? 
> 
> Current Distributions:
> ...

 

Is you gentoo system from this list?  Gentoo 2008.0 (i386 and x86_64)??  2008 is a while back, but that said, I'm sure it's perfectly competent.  Otherwise, you created it in the gentoo way?

 *Quote:*   

> 
> 
> I am keeping the filesystem-cloned Gentoo xen guest (which has not booted yet) 
> 
> 

 

gives me another clue to what you have.  The ubuntu lucid then is a host on your regular computer, or another vm that can communicate with the gentoo vm? I chroot from one vm to another regularly.

So, copying.  That is, copying the whole gentoo host to an image file, as in

make file.img

mkfs -t ... file.img

mount o loop file.img /mnt/loop

cp -a / /mnt/loop

or

tar -... / /tmp/gentoosys.tar.bz ??

tar [explode] /tmp/gentoosys.tar.bz /mnt/loop ??

This will give me the handle I need to move ahead.

Now, I shall go and read this extra link content while you find and digest all the above.

In anticipation of your outcome, PVgrub uses a pygrub approach to booting a guest.

There are two distinct approaches to booting para-virt, and I won't go into them until I'm clear on the desired setup. Once I'm clear, I can recommend you s kernel design.

You must be wanting to boot a gentoo 64 vm directly from a host linux on your computer, which has to be in a xen environment.  That I can get you to do.

----------

## jeffk

This might simplify the picture:

server2 is a physical production machine at my colocated facility with a static IP. It's running Gentoo ~amd64, no virtualization, gentoo-sources, genkernel:

```
server2 ~ $ uname -a

Linux server2 2.6.34-gentoo-r1 #1 SMP Fri Jun 18 07:31:22 EDT 2010 x86_64 Intel(R) Core(TM)2 Duo CPU E8500 @ 3.16GHz GenuineIntel GNU/Linux

server2 ~ $ head -n 2 /etc/make.conf

CHOST="x86_64-pc-linux-gnu"

CFLAGS="-march=core2 -O2 -pipe"
```

I want to clone this Gentoo to my linode, I copied it and set the goal of gettin it to boot as a Linode-Xen guest.

server2 filesystem was cloned to linode disk /dev/xvdc. I access this clone using a chroot within Ubuntu 10.04. Ubuntu 10.04 is a linode xen guest. I'm just running Ubuntu as a maintenance tool until I get the server 2 clone to boot as a Linode-Xen guest under its own genkernel using PV_GRUB.

```
root@mylinode:~# uname -a

Linux mylinode 2.6.32.12-x86_64-linode12 #1 SMP Wed Apr 28 18:47:37 UTC 2010 x86_64 GNU/Linux

root@mylinode:~# mount

/dev/xvda on / type ext3 (rw,noatime,errors=remount-ro)

proc on /proc type proc (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)

none on /sys type sysfs (rw,noexec,nosuid,nodev)

none on /sys/fs/fuse/connections type fusectl (rw)

/dev on /dev type devtmpfs (rw)

none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)

none on /dev/shm type tmpfs (rw,nosuid,nodev)

none on /var/run type tmpfs (rw,nosuid,mode=0755)

none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)

none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)

/dev/xvdc on /mnt/gentoo type ext3 (rw)

none on /mnt/gentoo/proc type proc (rw)

/dev on /mnt/gentoo/dev type none (rw,bind)

(chroot) mylinode # uname -a

Linux mylinode 2.6.32.12-x86_64-linode12 #1 SMP Wed Apr 28 18:47:37 UTC 2010 x86_64 Intel(R) Xeon(R) CPU L5520 @ 2.27GHz GenuineIntel GNU/Linux

(chroot) mylinode # head -n 2 /etc/make.conf

CHOST="x86_64-pc-linux-gnu"

CFLAGS="-march=nocona -O2 -pipe"

(chroot) mylinode #  # From here I can do stuff like emerge, configure, etc.

```

----------

## idella4

jeffk.

ok, good,  we will get there as I said.  This is 'nice' for both of us.  It introduces me to a real live xen application beyond what I've come across before and I can help you get this to boot.

So, you have made a fully fledged gentoo in the gentoo way.

The gentoo 2008 you have not utilised.  You want to use your gentoo vm as your choice of linode.

The ubuntu lucid, was that a stock vm acquired from linode?

ok, from here we can move ahead.  To help complete what I can't quite grasp, not having come across linode before now, how do you get to 

  root@mylinode:~# ?

From your gentoo 64, yes, but is it all to do with linode provided software that comes with your account?

Output of root@mylinode:~# mount

tells me that mylinode is in fact a xen vm.  lucid is a vm, let me guess, on linode's mount it would be /dev/xvdb.

You have indeed cloned your gentoo effectively as a vm which is accessed by mylinode.

It would be interesting to see mylinode's ls /boot, and lucid's ls /boot.

Now, PV-grub works as pygrub.  Step 1 may be the hardest, I've had trouble with it.  Chrooted within your gentoo64 vm, you need to install grub into the vm's 'mbr'.

In case you are impatient, the kernel, now be prepared to  have to make more than the first attempt to achieve success.  By rights, you don't need and xen source kernel.   PV-grub firstly reads the boot data in the mbr of the guest vm.  It then proceeds to boot the resident kernel, via finding menu.lst.  This means it should have trouble booting your lucid, because ubuntu uses grub2 which has replaced menu.lst with grub.cfg  [That would be adjusted in the initial creation of the vm by xen specific hooks].  That aside, the kernel prep guide of your link is on target.  You need a kernel that will boot a guest that supports a para-virt capable boot, hence the list provided for your kernel.

If you look at the gentoo wiki and docs, you will also find a gentoo made guide for compiling kernels for xen.

It instructs on making a dom0 and a domU kernel.  You want the domU kernel.

Now, here's the ace under my sleeve.  In the past, kernel configs were provided by ubuntu and fedora and still currently centos.  While the list of xen centric distros is shortening, we don't go without.  Ubuntu, having abandoned xen dom0, made a xen domU, which I found and ended up using. My attempts to make a kernel succeeded for booting a host, but wasn't quite there for booting guests.  It's a 2.6.31.  First, make a kernel with the settings from the link, acquire mkinitrd, and make an initrd along with the kernel.  You can obviously do all this within your chroot state in the gentoo vm since it WOULD have the kernel in place.  Place in /boot, and done.

Now, back to step 1.  A standard install of a vm incorporates installing a grub into the vm.  Your cloned gentoo guest is yet without this.  Your cloned gentoo guest will have /dev/xvda as /.  If you've made a separate /boot gentoo style, undo it.  in chroot, you have fdisk or parted, so no problem.    By rights, grub-install /dev/xvda.  I tried it and had real trouble.  To help it along, bring up your /boot/grub/device.map via gentoo style nano.  It will be in the state of your gentoo host 'machine'.  e.g. my current,

```

idella@gentoo64 ~ $ cat /boot/grub/device.map

(fd0)   /dev/fd0

(hd0)   /dev/sda

(hd1)   /dev/sdb

(hd2)   /dev/sdc

(hd3)   /dev/sdd

(hd4)   /dev/sde

```

You will need to adjust (hd0)   /dev/sda to    (hd0)   /dev/xvda.  Now remember once chroot into gentoo guest, you have done the gentoo style

mount -t proc none /mnt/gentpp/proc

mount /dev -o bind /mnt/gentoo/dev

So, from mylinode, /dev/xvda is /.  Once chroot into mnt/gentoo, gentoo vm will want /dev/xvda as root.

grub-install /dev/xvda will overwrite mylinode with gentoo vm content.  So grub-install /dev/xvda need be grub-install /dev/xvdc.

Oh, back to step 2.  This will give you an insight into my tinkering with xen ubuntu-2.6.31-domU; Go get it. 

You know, you may be close to all set.  By rights, PV-grub could well boot your gentoo clone with a genkernel.

You will have to try it out to see,  You need to get grub installed before you can attempt any kernel.  Try the above, then re-post. 

 Rest assured, no-one else browsing this section will be able to support your query. You will in the end post for me SUCCESS  :Laughing: 

That essay is enough.  Just one p.s.  Would you like to setup your gentoo host as xen capable and as a pre-step or an added extra, so you can boot your gentoo clone in your gentoo host?

----------

## jeffk

Just a quick reply to this thread that I completed the physical Gentoo to Linode Gentoo migration successfully. Thanks to all who helped. It was completed late last night, but I will post an overview of what was involved in the migration here soon.

Thanks again.

----------

## kimmie

Congrats. That linode doco looked very helpful - I hadn't come across pv-grub before. Even if you can't be bothered posting the whole shebang, I'm curious to know whether you went with a xen-sources or other kernel, and what your config was.

----------

## idella4

right, tie off the tread with [resolved] in your thread title.  PV-grub is a form of pygrub.

```

bash-3.2# ls /mnt/genny/usr/lib/xen/boot/

hvmloader  ioemu-stubdom.gz  pv-grub-x86_32.gz  pygrub

```

----------

## jeffk

Here is a preliminary list of the changes that were actually necessary to complete migration of my physical Core2Duo Gentoo ~amd64 machine to a Linode Xen guest VPS.

These steps follow a preliminary copy of the entire filesystem under /, excluding proc and dev. You should be able to chroot into /mnt/gentoo from some other Xen guest such as Ubuntu or Finnix, the following steps require some chroot or rescue environment. I decided to rebuild everything on the new Xen host after changin -march in /etc/make.conf:

```
-CFLAGS="-march=core2 -O2 -pipe"

+CFLAGS="-march=nocona -O2 -pipe"
```

For my custom kernel, I did not need to use xen-sources, which I think is for Xen hosts, not (necessarily) Xen guest. I stayed with gentoo-sources. Per the linode docs, I modified the existing genkernel configuration to include the following:

```
CONFIG_PARAVIRT_GUEST=y

CONFIG_XEN=y

CONFIG_PARAVIRT=y

CONFIG_PARAVIRT_CLOCK=y

CONFIG_XEN_BLKDEV_FRONTEND=y

CONFIG_XEN_NETDEV_FRONTEND=y

CONFIG_HVC_XEN=y

CONFIG_XEN_BALLOON=y

CONFIG_XEN_SCRUB_PAGES=y
```

Setting these in menuconfig creates an effective kernel diff (in my case) as follows:

```
diff --git a/kernel-config-x86_64-2.6.34-gentoo-r2 b/kernel-config-x86_64-2.6.34-gentoo-r2

index 624225a..7f567be 100644

--- a/kernel-config-x86_64-2.6.34-gentoo-r2

+++ b/kernel-config-x86_64-2.6.34-gentoo-r2

@@ -1,7 +1,7 @@

 #

 # Automatically generated make config: don't edit

-# Linux kernel version: 2.6.34-gentoo-r1

-# Fri Jun 18 07:13:06 2010

+# Linux kernel version: 2.6.34-gentoo-r2

+# Tue Jul 13 14:17:54 2010

 #

 CONFIG_64BIT=y

 # CONFIG_X86_32 is not set

@@ -251,7 +251,17 @@ CONFIG_X86_EXTENDED_PLATFORM=y

 # CONFIG_X86_VSMP is not set

 CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y

 CONFIG_SCHED_OMIT_FRAME_POINTER=y

-# CONFIG_PARAVIRT_GUEST is not set

+CONFIG_PARAVIRT_GUEST=y

+CONFIG_XEN=y

+CONFIG_XEN_MAX_DOMAIN_MEMORY=32

+CONFIG_XEN_SAVE_RESTORE=y

+# CONFIG_XEN_DEBUG_FS is not set

+# CONFIG_KVM_CLOCK is not set

+# CONFIG_KVM_GUEST is not set

+CONFIG_PARAVIRT=y

+# CONFIG_PARAVIRT_SPINLOCKS is not set

+CONFIG_PARAVIRT_CLOCK=y

+# CONFIG_PARAVIRT_DEBUG is not set

 CONFIG_NO_BOOTMEM=y

 # CONFIG_MEMTEST is not set

 # CONFIG_M386 is not set

@@ -869,7 +879,7 @@ CONFIG_FIRMWARE_IN_KERNEL=y

 CONFIG_EXTRA_FIRMWARE=""

 # CONFIG_DEBUG_DRIVER is not set

 # CONFIG_DEBUG_DEVRES is not set

-# CONFIG_SYS_HYPERVISOR is not set

+CONFIG_SYS_HYPERVISOR=y

 # CONFIG_CONNECTOR is not set

 # CONFIG_MTD is not set

 # CONFIG_PARPORT is not set

@@ -899,6 +909,7 @@ CONFIG_BLK_DEV_LOOP=y

 # CONFIG_BLK_DEV_RAM is not set

 # CONFIG_CDROM_PKTCDVD is not set

 # CONFIG_ATA_OVER_ETH is not set

+CONFIG_XEN_BLKDEV_FRONTEND=y

 # CONFIG_VIRTIO_BLK is not set

 # CONFIG_BLK_DEV_HD is not set

 CONFIG_MISC_DEVICES=y

@@ -1117,6 +1128,7 @@ CONFIG_R8169_VLAN=y

 # CONFIG_NET_PCMCIA is not set

 # CONFIG_WAN is not set

 # CONFIG_ATM_DRIVERS is not set

+CONFIG_XEN_NETDEV_FRONTEND=y

 # CONFIG_FDDI is not set

 # CONFIG_HIPPI is not set

 # CONFIG_PPP is not set

@@ -1145,6 +1157,7 @@ CONFIG_INPUT=y

 # CONFIG_INPUT_JOYDEV is not set

 CONFIG_INPUT_EVDEV=y

 # CONFIG_INPUT_EVBUG is not set

+CONFIG_XEN_KBDDEV_FRONTEND=y

 

 #

 # Input Device Drivers

@@ -1219,6 +1232,9 @@ CONFIG_SERIAL_CORE_CONSOLE=y

 CONFIG_UNIX98_PTYS=y

 # CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set

 # CONFIG_LEGACY_PTYS is not set

+CONFIG_HVC_DRIVER=y

+CONFIG_HVC_IRQ=y

+CONFIG_HVC_XEN=y

 # CONFIG_VIRTIO_CONSOLE is not set

 # CONFIG_IPMI_HANDLER is not set

 CONFIG_HW_RANDOM=y

@@ -1451,11 +1467,12 @@ CONFIG_FB_CFB_FILLRECT=y

 CONFIG_FB_CFB_COPYAREA=y

 CONFIG_FB_CFB_IMAGEBLIT=y

 # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set

-# CONFIG_FB_SYS_FILLRECT is not set

-# CONFIG_FB_SYS_COPYAREA is not set

-# CONFIG_FB_SYS_IMAGEBLIT is not set

+CONFIG_FB_SYS_FILLRECT=y

+CONFIG_FB_SYS_COPYAREA=y

+CONFIG_FB_SYS_IMAGEBLIT=y

 # CONFIG_FB_FOREIGN_ENDIAN is not set

-# CONFIG_FB_SYS_FOPS is not set

+CONFIG_FB_SYS_FOPS=y

+CONFIG_FB_DEFERRED_IO=y

 # CONFIG_FB_SVGALIB is not set

 # CONFIG_FB_MACMODES is not set

 # CONFIG_FB_BACKLIGHT is not set

@@ -1501,6 +1518,7 @@ CONFIG_FB_INTEL_I2C=y

 # CONFIG_FB_CARMINE is not set

 # CONFIG_FB_GEODE is not set

 # CONFIG_FB_VIRTUAL is not set

+CONFIG_XEN_FBDEV_FRONTEND=y

 # CONFIG_FB_METRONOME is not set

 # CONFIG_FB_MB862XX is not set

 # CONFIG_FB_BROADSHEET is not set

@@ -1796,6 +1814,16 @@ CONFIG_RTC_DRV_CMOS=m

 #

 # TI VLYNQ

 #

+

+#

+# Xen driver support

+#

+CONFIG_XEN_BALLOON=y

+CONFIG_XEN_SCRUB_PAGES=y

+CONFIG_XEN_DEV_EVTCHN=y

+CONFIG_XENFS=y

+CONFIG_XEN_COMPAT_XENFS=y

+CONFIG_XEN_SYS_HYPERVISOR=y

 # CONFIG_STAGING is not set

 CONFIG_X86_PLATFORM_DEVICES=y

 # CONFIG_ACER_WMI is not set
```

Xen-guest /etc/fstab will have a different disk scheme than /dev/sdaN:

```
# cat /etc/fstab 

# <fs>      <mountpoint>   <type>      <opts>      <dump/pass>

/dev/xvda         /               ext3            noatime         0 1

/dev/xvdb       none            swap            sw              0 0

shm      /dev/shm   tmpfs      nodev,nosuid,noexec   0 0

proc            /proc           proc      defaults        0 0 

/dev/ptmx   /dev/pts   devpts      defaults   0 0
```

Xen-guest PV-GRUB requires /boot/menu.lst, the following format worked for me:

```
# cat /boot/grub/menu.lst 

default 0

timeout 5

title Gentoo Linux 2.6.34-gentoo-r2

root (hd0)

kernel /boot/kernel-genkernel-x86_64-2.6.34-gentoo-r2 root=/dev/xvda
```

/etc/inittab needs an additional c0 terminal as follows, and then select No : Xenify Distro in your profile. If this was working correctly for me, Xenify would have added a similar line, I gather. Likewise with fstab, I think it would have tried to make the necessary changes there, but it didn't work for me.

```
c0:12345:respawn:/sbin/agetty 38400 hvc0 linux
```

In your linode profile, select kernel pv-grub-x86_64 (or 32, as appropriate).

I think that was it. The Linode Xen guest VPS boots now, there's still some details with /dev and ptys that I need to work out. Should I unmerge udev, etc.

Thanks.

----------

## kimmie

Thanks   :Smile: 

----------

