# howto install pvgrub for gentoo xen

## newtonian

From what I've seen posted in the xen forums, gentoo doesn't add pvgrub to their xen distribution because it uses downloads software from other packages.  There is pygrub useflag available for xen so you can use that.  Although pvgrub is supposed more secure.  

Peter Groben has created an overlay for pvgrub that worked very well for me.  This howto goes over the steps it took me to get pvgrub installed in a gentoo dom0 and the configs needed for the domU running  from a a gentoo-sources kernel compiled within the domU.

pvgrub really got me out of a tight spot for 2 reasons.  

 1. I couldn't re-compile my dom0 kernel because it is hosting several production domU's and I can't risk any downtime should the compile go wrong.

 2. With pvgrub domU's can freely build their own kernel and modules without ever having to upload anything to the dom0.

dom0 pvgrub, setup the overlay and build the package

Create a new directory that will be the root of your own portage tree, e. g. 

```
 mkdir /usr/local/portage
```

add the following to /etc/make.conf if you don't already have an overlay setup: 

```
echo 'PORTDIR_OVERLAY=/usr/local/portage' >> /etc/make.conf
```

create a xen-pvgrub directory 

```
mkdir -p /usr/local/portage/app-emulation/xen-pvgrub/
```

move to the xen-pvgrub folder and download the contents of the files directory and the ebuild you need ie.

```
cd /usr/local/portage/app-emulation/xen-pvgrub/

mkdir files
```

populate the files directory with patches:

```
cd /usr/local/portage/app-emulation/xen-pvgrub/files/

wget http://subversion.fem.tu-ilmenau.de/repository/fem-overlay/trunk/app-emulation/xen-pvgrub/files/xen-pvgrub-3.3.1-sandbox-fix.patch

wget http://subversion.fem.tu-ilmenau.de/repository/fem-overlay/trunk/app-emulation/xen-pvgrub/files/xen-pvgrub-3.3.2-32bit-ioemu-fix.patch

wget http://subversion.fem.tu-ilmenau.de/repository/fem-overlay/trunk/app-emulation/xen-pvgrub/files/xen-pvgrub-4.0.9999-sandbox-fix.patch

```

get the ebuild that you need

```
cd /usr/local/portage/app-emulation/xen-pvgrub/

wget http://subversion.fem.tu-ilmenau.de/repository/fem-overlay/trunk/app-emulation/xen-pvgrub/xen-pvgrub-3.3.2.ebuild

wget http://subversion.fem.tu-ilmenau.de/repository/fem-overlay/trunk/app-emulation/xen-pvgrub/xen-pvgrub-4.0.1.ebuild
```

create a digest for your ebuild ie.

```
ebuild xen-pvgrub-3.3.2.ebuild digest

ebuild xen-pvgrub-4.0.1.ebuild digest
```

emerge the package

```
echo 'app-emulation/xen-pvgrub' >> /etc/portage/package.keywords
```

```
emerge -av =xen-pvgrub-4.0.1
```

if all goes well, you should get the following results from equery:

```
equery files xen-pvgrub
```

```
/usr

/usr/lib

/usr/lib/xen

/usr/lib/xen/boot

/usr/lib/xen/boot/pv-grub-x86_32.gz

/usr/lib/xen/boot/pv-grub-x86_64.gz
```

updated settings for the domU:

```
cat /mnt/data/xen/configs/mydomU.conf
```

```
name   = "mydomU"

#pvgrub here

kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"

extra='(hd2)/boot/grub/grub.conf'

#end pvgrub

memory = 1024 

vcpus  = 4

disk = [ 'phy:virtual-servers/dev-root,xvda4,w', \

         'phy:virtual-servers/dev-swap,xvda2,w', \

    'phy:virtual-servers/dev-tmp,xvda3,w' ]

vif = [ "rate=750KB/s, mac=00:11:50:00:00:62, bridge=xenbr0" ]
```

domU settings with standard gentoo sources

Now we're done with the settings on the dom0 side.

From within domU, create a /boot/grub/grub.conf.  There is no need to emerge grub. pvgrub only reads the grub.conf text file.

```
mkdir -p /boot/grub/

vim /boot/grub/grub.conf
```

Sample grub.conf when using pvgrub and gentoo-sources in domU

```
default 0

timeout 5

title linux-2.6.36-gentoo-r5

root (hd2)

kernel /boot/linux-2.6.36-gentoo-r5 root=/dev/xvda4 ro

```

```
emerge gentoo-sources
```

create a new symlink

```
cd /usr/src/ rm linux; ln -s linux-2.6.36-gentoo-r5 linux 
```

setup the kernel

```
cd /usr/src/linux; make menuconfig
```

 *Quote:*   

> Processor type and features ---> Paravirtualized guest support ---> [*] Xen guest support
> 
> Device Drivers ---> Block Devices ---> [*] Xen virtual block device support
> 
> Device Drivers ---> Network device support ---> [*] Xen network device frontend driver
> ...

 

build the kernel, also were using pvgrub so it's OK to add modules to the kernel!

```
make -j5 && make modules_install
```

```
cp vmlinux /boot/linux-2.6.36-gentoo-r5
```

update fstab, change block device name like sda to xvda

```
sed -i 's/sda/xvda/' /etc/fstab
```

sample /etc/fstab:

```

/dev/xvda2               none            swap            sw                             0 0

/dev/xvda3               /tmp            tmpfs           nodev,nosuid,noexec   0 0

/dev/xvda4               /                  ext3            noatime                      0 1

shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0

```

In /etc/inittab change tty1 to hvc0 and add hvc0 to /etc/securetty

```
sed -i 's/tty1/hvc0/' /etc/inittab

echo 'hvc0' >> /etc/securetty
```

shutdown the domU and create the domU from the dom0 side.  You should get the 

```
my domUprompt # shutdown -h now
```

from the dom0 side create the new domU and have it boot into pvgrub:

```
xm create -c /mnt/data/xen/configs/mydomUconfig
```

```
   GNU GRUB  version 0.97  (1048576K lower / 0K upper memory)

 +-------------------------------------------------------------------------+

 | linux-2.6.36-gentoo-r5                                                  |  

 | Xen Linux dev-built-linux-2.6.34-xen-r4                                 |

 |                                                                         |

 |                                                                         |

 |                                                                         |

 |                                                                         |  

 +-------------------------------------------------------------------------+

    Use the ^ and v keys to select which entry is highlighted.

    Press enter to boot the selected OS, 'e' to edit the

    commands before booting, or 'c' for a command-line.
```

You should now be free to create and install new modules and kernels all within the domU filesystem.

Cheers,

Links:

https://bugs.gentoo.org/236380

http://www.linode.com/wiki/index.php/PV-GRUB#Building_Your_Own_Kernel

http://backdrift.org/xen-pvgrub-howto

https://forums.gentoo.org/viewtopic-t-840251-highlight-pygrub.html

http://lists.xensource.com/archives/html/xen-users/2010-09/msg00804.html

Last edited by newtonian on Tue Dec 28, 2010 1:00 pm; edited 1 time in total

----------

## aTan

Hi. Thank you for pvgrub howto.

When I'm trying to emerge xen-pvgrub I'm getting this error:

```
gcc -DCONFIG_GRUB   -fno-strict-overflow -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls  -m32 -march=i686 -m32 -march=i686 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement  -fno-stack-protector -fno-exceptions -fno-builtin -Wall -Werror -Wredundant-decls -Wno-format -Wno-redundant-decls -fno-stack-protector -fgnu89-inline -Wstrict-prototypes -Wnested-externs -Wpointer-arith -Winline -g -DGNT_DEBUG -DGNTMAP_DEBUG -D__INSIDE_MINIOS__ -m32 -march=i686 -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/../extras/mini-os/include -D__MINIOS__ -DHAVE_LIBC -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/../extras/mini-os/include/posix -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/../tools/xenstore  -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/../extras/mini-os/include/x86 -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/../extras/mini-os/include/x86/x86_32 -U __linux__ -U __FreeBSD__ -U __sun__ -nostdinc -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/../extras/mini-os/include/posix -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/cross-root-i686/i686-xen-elf/include -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.1/include -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/lwip-x86_32/src/include -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/lwip-x86_32/src/include/ipv4 -I/var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/include -I../xen/include -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/extras/mini-os/../../extras/mini-os/include -D__MINIOS__ -DHAVE_LIBC -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/extras/mini-os/../../extras/mini-os/include/posix -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/extras/mini-os/../../tools/xenstore -D__XEN_INTERFACE_VERSION__=0x00030205  -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/extras/mini-os/../../extras/mini-os/include/x86 -isystem /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/extras/mini-os/../../extras/mini-os/include/x86/x86_32 -c sched.c -o /var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/mini-os-x86_32-grub/sched.o

cc1: warnings being treated as errors

netfront.c:41:8216: error: variably modified ‘tx_freelist’ at file scope

netfront.c:44:8218: error: variably modified ‘rx_buffers’ at file scope

netfront.c:45:8218: error: variably modified ‘tx_buffers’ at file scope

make[1]: *** [/var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom/mini-os-x86_32-grub/netfront.o] Error 1

make[1]: *** Waiting for unfinished jobs....

make[1]: Leaving directory `/var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/extras/mini-os'

make: *** [pv-grub] Error 2

make: Leaving directory `/var/tmp/portage/portage/app-emulation/xen-pvgrub-4.0.1/work/xen-4.0.1/stubdom'

emake failed

 * ERROR: app-emulation/xen-pvgrub-4.0.1 failed:

 *   compile pv-grub_x86_32 failed

 * 

 * Call stack:

 *     ebuild.sh, line  56:  Called src_compile

 *   environment, line 2688:  Called die

 * The specific snippet of code:

 *       emake XEN_TARGET_ARCH="x86_32" -C stubdom pv-grub || die "compile pv-grub_x86_32 failed";
```

```
Portage 2.2.0_alpha10 (default/linux/amd64/10.0/desktop, gcc-4.5.1, glibc-2.12.1-r3, 2.6.36-zen1-bfs x86_64)

=================================================================

System uname: Linux-2.6.36-zen1-bfs-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_3800+-with-gentoo-2.0.1

Timestamp of tree: Sun, 26 Dec 2010 01:45:01 +0000

distcc 3.1 x86_64-pc-linux-gnu [disabled]

ccache version 3.1.3 [enabled]

app-shells/bash:     4.1_p9

dev-java/java-config: 2.1.11-r3

dev-lang/python:     2.6.6-r1, 2.7.1, 3.1.3

dev-util/ccache:     3.1.3

dev-util/cmake:      2.8.3-r1

sys-apps/baselayout: 2.0.1-r1

sys-apps/openrc:     0.6.8

sys-apps/sandbox:    2.4

sys-devel/autoconf:  2.13, 2.68

sys-devel/automake:  1.6.3-r1, 1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1

sys-devel/binutils:  2.21

sys-devel/gcc:       4.4.5, 4.5.1-r1

sys-devel/gcc-config: 1.4.1

sys-devel/libtool:   2.4-r1

sys-devel/make:      3.82

virtual/os-headers:  2.6.34 (sys-kernel/linux-headers)

Repositories: gentoo gnome zugaina rion kde-sunset kde sunrise local

ACCEPT_KEYWORDS="amd64 ~amd64"

ACCEPT_LICENSE="*"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-O2 -march=native -pipe -fomit-frame-pointer -fno-strict-overflow -fno-delete-null-pointer-checks -fno-tree-vrp -mno-align-stringops -minline-stringops-dynamically -mno-push-args -fno-ident"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config /usr/share/openvpn/easy-rsa"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo"

CXXFLAGS="-O2 -march=native -pipe -fomit-frame-pointer -fno-strict-overflow -fno-delete-null-pointer-checks -fno-tree-vrp -mno-align-stringops -minline-stringops-dynamically -mno-push-args -fno-ident"

DISTDIR="/usr/portage/distfiles"

EMERGE_DEFAULT_OPTS="--with-bdeps y --keep-going"

FEATURES="assume-digests binpkg-logs buildpkg ccache distlocks fixlafiles fixpackages metadata-transfer news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"

GENTOO_MIRRORS="ftp://ftp.linux.cz/pub/linux/gentoo"

LANG="en_US.UTF-8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

LINGUAS="en cs ru"

MAKEOPTS="-j2"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp/portage"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/var/lib/layman/gnome /var/lib/layman/zugaina /var/lib/layman/rion /var/lib/layman/kde-sunset /var/lib/layman/kde /var/lib/layman/sunrise /system/portage-local"

SYNC="rsync://rsync.de.gentoo.org/gentoo-portage"

USE="3dnow 3dnowext 64bit 7zip X a52 aac acl acpi alsa amd64 avi bash-completion bitmap-fonts bluetooth branding bzip2 cairo cddb cdr cli cpudetection cracklib crypt css cups cxx dbus devmap dlloader dri dts dv dvd dvdr dvdread emboss emul-linux-x86 encode exif extensions fam fbcon ffmpeg firefox flac fortran ftp fuse gdbm gdu gif glut gnutls gpm gstreamer gtk gtk2 htmlhandbook iconv ieee1394 ipv6 jpeg kde lame lcms libg++ libnotify lm_sensors logitech-mouse mad mikmod mmx mng modules mp3 mp4 mpeg mplayer mudflap multilib musepack musicbrainz ncurses nls nptl nptlonly nvidia ogg opengl openmp oss pam pango pcre pdf pdflib perl png policykit ppds pppd python qt3 qt3support qt4 quicktime readline reflection sdl semantic-desktop session spell spl sse sse-filters sse2 sse3 ssl startup-notification svg symlink sysfs tcpd threads tiff tk truetype truetype-fonts type1-fonts udev unicode usb utf8 vcd vorbis x264 xcb xcomposite xml xorg xscreensaver xulrunner xv xvid zlib" ALSA_CARDS="intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en cs ru" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 

Unset:  CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

```

----------

## aTan

Switching back to gcc-4.4 fixed the error.

----------

## aTan

Is it possible to use pygrub with nfsroot?

----------

## newtonian

 *aTan wrote:*   

> Is it possible to use pygrub with nfsroot?

 

The guy in this post seems to have done it in pygrub, I'd imagine pvgrub would be just about the same setup.

http://forums.citrix.com/thread.jspa?threadID=240826&tstart=45

HTH,

----------

## newtonian

 *aTan wrote:*   

> Is it possible to use pygrub with nfsroot?

 

Also in pvgrub you be changing the domU config in /mnt/data/xen/configs/ on dom0  

from something like this:

```
kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"

extra = "(hd0,0)/grub/menu.lst"
```

to something like this:

```
kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"

extra = "(nd)/grub/menu.lst"

```

You should setup the DHCP server and TFTP server correctly. Create ${tftproot}/grub/menu.lst; copy necessary files (vmlinuz, initrd, etc.). 

From: http://wiki.xensource.com/xenwiki/PvGrub

I haven't tested a pvgrub nfsroot boot but I'd assume the following:

 /usr/lib/xen/boot/pv-grub-x86_32.gz would be coming from the dom0 file system and the location of grub/menu.lst would be dependent on 

the ip address given by the dhcp server and the /etc/exports ip to file system mapping.

Hope this helps,

----------

## alexxy

Thanks for pvgrub ebuilds. I made fixes for then and going to add them to main tree

----------

## newtonian

 *alexxy wrote:*   

> Thanks for pvgrub ebuilds. I made fixes for then and going to add them to main tree

 

That was Peter Groben's excellent work on the ebuilds, so I can't take credit for them but great to hear they'll be added to the main tree.

Thanks!!

----------

