# Xen 4 and hardened paravirt domUs

## at_chaos

UPDATE 4: 18.8.2010 20:30 GMT

+ added bridged networking domU /etc/conf.d/net setup

+ changed domU kernelconfig -> Pax enabled but failing option CONFIG_PAX_KERNEXEC disabled

+ added documentataion sources

Hi, the discussion started on bugtracker #279795. The goal is to run paravirtualised Xen domUs with hardened-sources and hardened profile.

As of writing this it was not possible to start a domU with latest hardened-sources-2.6.32-r9. The only way to get it booting is to use Security Level -> Custom instead of Security Level -> server rbac disabled or other. Hopefully we can find a solution together and make a little howto.

Docs - this howto here is based on following docs:

http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml

http://www.gentoo.org/doc/en/xen-guide.xml

http://en.gentoo-wiki.com/wiki/Xen

http://wiki.xensource.com/xenwiki/FrontPage?action=show&redirect=StartSeite

Assumptions:

We build a 64bit headless xen-4 hypervisor, the hardened guests are headless 64bit too. If you want to build 32bit support and/or graphical output check the gentoo-wiki http://en.gentoo-wiki.com/wiki/Xen

I do not want to cover all possibilities as they may confuse more than help.

Disk /dev/sda:

/dev/sda1 is our /boot partition, ext2

/dev/sda2 is our swap partition

/dev/sda3 is our root partition

/dev/sda4 holds a lvm volume group, not needed here

(I did a raid 1, lvm2 install but I do not cover this here as it would be too confusing) 

Store of xen stuff:

/etc/xen --> xend configuration files 

/mnt/xen/configs --> my xen domU configuration files folder

/mnt/xen/kernels --> my xen domU kernel folder

/mnt/xen/vms --> my xen domU image files folder

Networking

With xen we cover A) bridged networking (default) and B) routed network

Networking ips

Legend: 

dom0 ip: ddd.ddd.ddd.ddd

domU ip: uuu.uuu.uuu.uuu

gateway: rrr.rrr.rrr.rrr

nameserver1: nnn.nnn.nnn.nnn

nameserver2: mmm.mmm.mmm.mmm 

netmask: kkk.kkk.kkk.kkk

HowTo

dom0 Hypervisor

1) Prepare base system

boot livecd, partition your disks and create filesystem, see official handbook

http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=1

1.1) Mount partitions

```
# mkdir /mnt/gentoo

# mount /dev/sda3 /mnt/gentoo

# mkdir /mnt/gentoo/boot

# mount /dev/sda1 /mnt/gentoo/boot

# cd /mnt/gentoo
```

1.2) get stage3 from a gentoo mirror near you

```
# links http://www.gentoo.org/main/en/mirrors.xml
```

choose a mirror near you

download releases/amd64/autobuilds/current-stage3/stage3-amd64-DATE.tar.bz2

download releases/amd64/autobuilds/current-stage3/stage3-amd64-DATE.tar.bz2.CONTENT

download releases/amd64/autobuilds/current-stage3/stage3-amd64-DATE.tar.bz2.DIGEST

1.3) get latest portage tree

download snapshots/portage-latest.tar.bz2

download snapshots/portage-latest.tar.bz2.md5sum

1.4) verify stage3

```
# md5sum -c stage3-amd64-DATE.tar.bz2.DIGEST
```

1.5) extract stage3

```
# tar xvjf stage3-amd64-DATE.tar.bz2 -C /mnt/gentoo

```

1.6) verify portage-latest

```
# md5sum -c portage-latest.tar.bz2.md5sum
```

extract portage

```
# tar xvjf portage-latest.tar.bz2 -C /mnt/gentoo/usr
```

1.7) adjust /etc/make.conf

 *Quote:*   

> CFLAGS="-march=native -O2 -pipe"
> 
> CXXFLAGS="${CFLAGS}"
> 
> ## WARNING: Changing your CHOST is not something that should be done lightly.
> ...

 

1.8) copy /etc/resolv.conf

```
# cp -L /etc/resolv.conf /mnt/gentoo/etc/
```

1.9) mount proc and dev

```
# mount -t proc none /mnt/gentoo/proc

# mount -o bind /dev /mnt/gentoo/dev
```

1.10) chroot

```
# chroot /mnt/gentoo /bin/bash

# env-update

# source /etc/profile

# export PS1="(dom0-chroot) $PS1"
```

1.11) sync portage

```
# emerge --sync
```

1.12) Choose profile

show available profiles the profile marked by * is the current selected

```
# eselect profile list
```

output:

 *Quote:*   

>   [1]   default/linux/amd64/10.0 *
> 
>   [2]   default/linux/amd64/10.0/desktop
> 
>   [3]   default/linux/amd64/10.0/desktop/gnome
> ...

 

(alternative) you can also use

```
# eselect profile show

```

we want option [7] default/linux/amd64/10.0/server profile 

```
# eselect profile set [7]
```

double check if the right profile was set

```
# eselect profile show
```

1.13) set locales

```
# nano -w /etc/locale.gen

# locale-gen
```

1.14) set your timezone (choose your timezone in /usr/share/zoneinfo)

```
# cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime

```

2) Installing Xen and Xen kernel

2.1) Set xen related useflags

as we may also want hvm support (headless tough) we have to set that use flags for xen-tools, did not test it but pae is likely not needed on 64bit systems

```
# mkdir /etc/portage

# nano -w /etc/portage/package.use
```

 *Quote:*   

> app-emulation/xen-tools hvm 
> 
> app-emulation/xen pae

 

2.2) we need to unmask xen-4

```
# nano -w /etc/portage/package.keywords
```

 *Quote:*   

> app-emulation/xen
> 
> app-emulation/xen-tools
> 
> sys-kernel/xen-sources
> ...

 

2.3) get xen stuff

# emerge xen xen-tools xen-sources -av

output(R should be N on your system):

 *Quote:*   

> [ebuild   R   ] app-emulation/xen-tools-4.0.0  USE="hvm -acm -api -custom-cflags -debug -doc -flask -ioemu -pygrub -screen" 0 kB
> 
> [ebuild   R   ] sys-kernel/xen-sources-2.6.34  USE="-build -deblob -symlink" 0 kB
> 
> [ebuild   R   ] app-emulation/xen-4.0.0  USE="pae -acm -custom-cflags -debug -flask -xsm" 0 kB
> ...

 

2.4) add xend to default runlevel

```
# rc-update add xend default
```

2.5) Configure Xen dom0 kernel

```
# cd /usr/src/linux-2.6.34-xen
```

2.5.1a ) download my dom0 .config and adjust to your hardware

Configuration dom0 xen-sources-2.6.34:

```
# wget http://pastebin.ca/raw/1917417

# mv 1917417 .config
```

skip 2.5.1b) and goto 2.5.2) build kernel

2.5.1b) manual configuration see gentoo-wiki: 

http://en.gentoo-wiki.com/wiki/Xen#Domain_0_Kernel_Configuration

2.5.2) Build kernel

```
# make
```

2.5.6) Copy kernel image to /boot

```
# cp vmlinux /boot/vmlinuz-2.6.34-dom0
```

2.6) configure /etc/fstab

see also gentoo handbook http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=1&chap=8

```
# nano -w /etc/fstab
```

 *Quote:*   

> /dev/sda1               /boot           ext2            noauto,noatime  1 2
> 
> /dev/sda3               /               ext4            noatime         0 1
> 
> /dev/sda2               none            swap            sw              0 0
> ...

 

3) Networking dom0

Legend: 

dom0 ip: ddd.ddd.ddd.ddd

domU ip: uuu.uuu.uuu.uuu

gateway: rrr.rrr.rrr.rrr

nameserver1: nnn.nnn.nnn.nnn

nameserver2: mmm.mmm.mmm.mmm 

netmask: kkk.kkk.kkk.kkk

3.1) Set dom0 hostname

```
# nano -w /etc/conf.d/hostname
```

 *Quote:*   

> HOSTNAME="xen"

 

3.2) Set dom0 Domainname and network configuration

depends on your network infrastructure

see gentoo handbook http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4

```
# nano -w /etc/conf.d/net
```

A) BRIDGED SETUP

 *Quote:*   

> dns_domain="example.tld"
> 
> config_eth0=( "ddd.ddd.ddd.ddd netmask kkk.kkk.kkk.kkk" )
> 
> routes_eth0=( "default via rrr.rrr.rrr.rrr" )
> ...

 

B) ROUTED SETUP

 *Quote:*   

> dns_domain_lo="example.tld"
> 
> modules=("iproute2")
> 
> config_eth0=( "ddd.ddd.ddd.ddd/27 peer rrr.rrr.rrr.rrr" )
> ...

 

3.3) Add eth0 to default runlevel

```
# rc-update add net.eth0 default

```

3.4) edit hosts file

```
# nano -w /etc/hosts
```

 *Quote:*   

> 127.0.0.1       xen.example.tld xen localhost
> 
> ::1             xen.example.tld xen localhost

 

4) Networking Xen

The official gentoo xen howto has a nice description how to configure bridged and routet network setup.

http://www.gentoo.org/doc/en/xen-guide.xml#doc_chap4

5) Other system configuration

5.1) Set root password

```
# passwd
```

5.2) Set keymap

```
# nano -w /etc/conf.d/keymaps
```

5.3) Set clock

```
# nano -w /etc/conf.d/clock
```

5.4) Install system tools (syslog, cron), see http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=1&chap=9

5.5) SSHD

uncomment PermitRootLogin if you want to be able to log in as root (if it works you should switch to key auth on production server)

```
# nano -w /etc/ssh/sshd_config
```

 *Quote:*   

> PermitRootLogin yes 

 

5.6) add it to default runlevel

```
# rc-update add sshd default
```

6) Grub Bootloader

6.1) Install grub, see http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=1&chap=10

6.2) Configure grub to startup our xen kernel

```
# nano -w /boot/grub/grub.conf
```

 *Quote:*   

> default 0
> 
> timeout 10
> 
> title Xen 4.0 / Linux 2.6.34
> ...

 

Note: If your server hangs on rebooting the xen kernel try to add acpi=off to the bootoptions

Our dom0 is now finished. >ou can now reboot to check if your xen kernel works or you can go ahead and configure your hardened domU kernel and reboot after that, your choice.

7) Configuring Hardened DomU kernel

7.1) get hardened sources and go to sources

```
# emerge hardened-sources

# cd /usr/src/linux-2.6.32-hardened-r9
```

7.2a) Configure the kernel with xen support or take a copy of my config

Configuration file of PV domU hardened-sources-2.6.32-r9

```
# wget http://pastebin.ca/raw/1919262

# mv 1919262 .config
```

skip 7.2b) and goto 7.3) build kernel

7.2b) Configure your kernel skip this if you copied above mentioned config 

```
# make menuconfig
```

In menuconfig enable xen features:

[quote]Processor type and features ---> Paravirtualized guest support ---> [*] Xen guest support

Device Drivers ---> Block Devices ---> [*] Xen virtual block device support

Device Drivers ---> Network device support ---> [*] Xen network device frontend driver

Device Drivers ---> [*] Xen memory balloon driver

                             [*] Scrub pages before returning them to system

                             [*] Xen /dev/xen/evtchn device

                             [*] Xen filesystem

                             [*] Create xen entries under /sys/hypervisor

Still in menuconfig you go to

Security Options ---> Grsecurity ---> Security Level ---> (X) Hardened Gentoo [server no rbac]

(this will enable all needed grsecurity and PaX options for you)

Because the domU does not start with this security level but we want all the good stuff enabled we have to switch to

Security Options ---> Grsecurity ---> Security Level ---> (X) Custom

Exit menuconfig and save the configuration

7.3) Build kernel

```
# make
```

7.4) Copy it to our xen kernel folder

```
# cp vmlinux "/mnt/xen/kernels/gentoo-hardened-2.6.32-r9

```

We are now finished with the preparation on dom0. If you did not reboot before building the hardened domU kernel. You should do that now.

DomU Hardened Guest(s)

1) Basic System setup

1.1) create lvm volume or partition or image file

1.2) mount domu lvm volume or physical partition or image file 

```
# mkdir /mnt/domu1

# mount /dev/virt/domu1 /mnt/domu1

# cd /mnt/domu1
```

1.3) get hardened stage3 from a gentoo mirror near you

```
# links http://www.gentoo.org/main/en/mirrors.xml
```

choose a mirror near you

download /releases/amd64/autobuilds/LATESTDATE/hardened/stage3-amd64-hardened-LATESTDATE.tar.bz2 (LATESTDATE is the latest folder e.g. 20100812)

and /releases/amd64/autobuilds/LATESTDATE/hardened/stage3-amd64-hardened-LATESTDATE.tar.bz2.CONTENTS

and /releases/amd64/autobuilds/LATESTDATE/hardened/stage3-amd64-hardened-LATESTDATE.tar.bz2.DIGEST

1.4) get latest portage tree

download snapshots/portage-latest.tar.bz2

and snapshots/portage-latest.tar.bz2.md5sum

1.5) verify stage3 download

```
# md5sum -c stage3-amd64-hardened-LATESTDATE.tar.bz2.DIGEST

```

1.6) extract hardened-stage3

```
# tar xvjf stage3-amd64-hardened-LATESTDATE.tar.bz2
```

1.7) verify portage-latest download

```
# md5sum -c portage-latest.tar.bz2.md5sum
```

1.8) extract portage

```
# tar xvjf portage-latest.tar.bz2 -C usr/
```

1.9) copy /etc/make.conf from dom0 and adjust it

```
# cp /etc/make.conf /mnt/domu1/etc/
```

make sure to adjust MAKEOPTS to your assigned cpus (ruleofthumb cpu cores + 1)

```
# nano -w /mnt/domu1/etc/make.conf
```

 *Quote:*   

> MAKEOPTS="-j3"

 

1.9) copy /etc/resolv.conf

```
# cp -L /etc/resolv.conf /mnt/domu1/etc/
```

1.10) mount proc and dev

```
# mount -t proc none /mnt/domu1/proc

# mount -o bind /dev /mnt/domu1/dev
```

1.11) chroot

```
# chroot /mnt/domu1 /bin/bash

# env-update

# source /etc/profile

# export PS1="(domU-chroot) $PS1"
```

1.12) sync portage

```
# emerge --sync
```

1.13) profile

show available profiles and check if the hardened profile is selected (it should if you use hardend-stage3), marked by *

```
# eselect profile list
```

 *Quote:*   

>   [1]   default/linux/amd64/10.0 
> 
>   [2]   default/linux/amd64/10.0/desktop
> 
>   [3]   default/linux/amd64/10.0/desktop/gnome
> ...

 

(alternative) you can also use

```
# eselect profile show
```

(optional) if you want another hardened profile -> choose it by setting the number displayed in front of the profile list output above

```
# eselect profile set [8]
```

double check if the right profile was set

```
# eselect profile show
```

1.14) set locales

```
# nano -w /etc/locale.gen

# locale-gen
```

1.15) set your timezone (choose your timezone in /usr/share/zoneinfo)

```
# cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime
```

1.16) edit /etc/fstab (see also gentoo handbook)

we assume that we name our root partition xvda1 and the swap partition xvda2 in our domU-xen-config (we will do that later)

```
# nano -w /etc/fstab
```

 *Quote:*   

> 
> 
> /dev/xvda1              /               ext4            noatime     		    0 1
> 
> /dev/xvda2              none            swap            sw              		0 0
> ...

 

2) Xen domU Networking

2.1) Set domU hostname

```
# nano -w /etc/conf.d/hostname
```

 *Quote:*   

> HOSTNAME="domu1"

 

2.2) Set domU Domainname

```
# nano -w /etc/conf.d/net
```

 *Quote:*   

> dns_domain_lo="example.tld"

 

2.3) Network configuration

Legend: 

dom0 ip: ddd.ddd.ddd.ddd

domU ip: uuu.uuu.uuu.uuu

gateway: rrr.rrr.rrr.rrr

nameserver1: nnn.nnn.nnn.nnn

nameserver2: mmm.mmm.mmm.mmm 

netmask: kkk.kkk.kkk.kkk

Bridged or routed setup? This depends on your network infrastructure and what you selected on xend setup (see dom0 howto). If you have bridged xen network setup A) or a routed network setup B). You can even use other methods like dhcp or nat but this is out of scope here. 

See xen docs (section routing): http://wiki.xensource.com/xenwiki/XenNetworking

```
# nano -w /etc/conf.d/net
```

2.3.1A) Bridged setup 

 *Quote:*   

> config_eth0=( "uuu.uuu.uuu.uuu netmask kkk.kkk.kkk.kkk" )
> 
> routes_eth0=( "ddd.ddd.ddd.ddd" )
> 
> dns_servers_eth0="nnn.nnn.nnn.nnn mmm.mmm.mmm.mmm"
> ...

 

2.3.1B) Routed setup

# Basically we make a pointopoint connections between the dom0 and domU(s) and dom0 is the gateway for domU(s). This is based on the www.hetzner.de datacenter network and included additional ips - other setups may differ

 *Quote:*   

> config_eth0=( "uuu.uuu.uuu.uuu netmask kkk.kkk.kkk.kkk pointopoint ddd.ddd.ddd.ddd" )
> 
> routes_eth0=( "ddd.ddd.ddd.ddd" )
> 
> dns_servers_eth0="nnn.nnn.nnn.nnn mmm.mmm.mmm.mmm"
> ...

 

2.3.2) add eth0 to default runlevel

```
# rc-update add net.eth0 default
```

2.3.4) edit /etc/hosts

```
# nano -w /etc/hosts
```

 *Quote:*   

> 127.0.0.1		domu1.example.tld domu1 localhost

 

3) Other System Config

3.1) set root password

```
# passwd
```

3.2) Keymap setup

More info about the following keymap and clock setup on offical handbook:

http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=1&chap=8

choose your keymap

```
# nano -w /etc/conf.d/keymaps
```

3.2) set clock

```
# nano -w /etc/conf.d/clock
```

#####todo check hw-clock error, minor problem

3.3) Install system tools (syslog, cron, ...), see official handbook:

http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=1&chap=9

3.4) SSH

uncomment PermitRootLogin if you want to be able to log in as root, you should disable it and switch to key auth if everything works

```
# nano -w /etc/ssh/sshd_config
```

 *Quote:*   

> PermitRootLogin yes 

 

add it to default runlevel

```
# rc-update add sshd default
```

3.5) to make xen console working with our hardened system 

```
# nano -w /etc/inittab
```

add to SERIAL part

 *Quote:*   

> h0:12345:respawn:/sbin/agetty 9600 hvc0 screen

 

```
# nano -w /etc/securetty
```

add hvc0 to the bottom

 *Quote:*   

> hvc0

 

3.6) We are done in the chroot. Exit and umount

```
# exit

# cd

# umount /mnt/domu1/proc

# umount /mnt/domu1/dev

# umount /mnt/domu1

```

4) Hardened paravirt configuration

Now we can configure our hardened domU

assuming we store our xen domU configs in /mnt/xen/configs

```
# nano -w /mnt/xen/configs/domu1.pv
```

 *Quote:*   

> kernel = "/mnt/xen/kernels/gentoo-hardened-2.6.32-r9"
> 
> memory = 2048
> 
> name = "domu1"
> ...

 

4.1) If all is set and the paths are correct we can start the domU

```
# xm create /mnt/xen/configs/domu1.pv -c
```

Last edited by at_chaos on Wed Aug 18, 2010 7:38 pm; edited 10 times in total

----------

## 229566

I'm interested in this, since I'd like to run hardened under a Xen VPS.

----------

## idella4

Ratrace,

it will take me a while to digest this fully, but to start with,

```

CFLAGS="-march=native -pipe -O2 -mno-tls-direct-seg-refs"

```

straight from, the gentoo xen wiki, re-compile world.

The merge info hasn't  stipulated the version of xen & xen-tools, could you post that.  gentoo's packages have issues.

Take it from there.\

I'm actually in the process of preparing a hardened gentoo vm so I can likely parallel test your issue

----------

## at_chaos

Hi,

I started yesterday to setup a new dom0 from scratch as well as a hardened domU. I take notes and will then have a walkthrough to post here which others can follow.

@idella4

This is a only needed for 32bit systems. All the notes around in the wiki are a little bit confusing. Hope to make it clearer in the upcoming walkthrough.

From http://en.gentoo-wiki.com/wiki/Xen#TLS_and_CFLAGS

 *Quote:*   

> Note: The '-mno-tls-direct-seg-refs' flag does not make sense on any 64bit system. For such systems you can skip the recompilation of the whole world and just recompile glibc

 

----------

## idella4

at_chaos

 *Quote:*   

> 
> 
>  it was not possible to start a domU with latest hardened-sources-2.6.32-r9
> 
> 

 

I take it you are using pygrub to boot a gentoo vm with the hardened kernel.

Can you cite& post  the error of the vm failing boot?

Do you have any other vms at the moment?

 *Quote:*   

> 
> 
> This is a only needed for 32bit systems.
> 
> 

 

i.e. a 32 bit gentoo gust, in which case it will need the -mno-tls-direct-seg-refs flag.

The current gentoo xen ebuild is 4.0.0.  Is this your xen hypervisor?  If so, not surprised.  Waiting for your reply.

----------

## Elbryan

I confirm that those settings work in a 32-bit system.

I made that kernel working disabling PAX on my Intel Atom (that doesn't have HVM capabilities). Great!

----------

## Elbryan

 *idella4 wrote:*   

> 
> 
> i.e. a 32 bit gentoo gust, in which case it will need the -mno-tls-direct-seg-refs flag.
> 
> The current gentoo xen ebuild is 4.0.0.  Is this your xen hypervisor?  If so, not surprised.  Waiting for your reply.

 

Do you mean that a Gentoo 32-bit guest needs that flag too? I have it only enabled in my dom0.

----------

## idella4

Elbryan,

I should double check but I would say yes.  If the guest is to be booted paravirt be a xen kernel, then I'd say  it should be.  If it's booted by pygrub which boots a resident regular kernel, then it makes sense not.

Ah I remember now.  When I was building the gentoo guest in paravirt mode, in building the vm up, then emerge itself observe it's a guest in xen and prompted to set the flag.  i.e. guest has no  kernel, booted by the xen guest kernel, resident on the host.

Do you have xen-4.0 working?  Mine's broken

----------

## 229566

 *idella4 wrote:*   

> Ratrace,
> 
> it will take me a while to digest this fully, but to start with,
> 
> ```
> ...

 

I followed your* example for domU kernel setup, basically disabling PaX and I can boot fine via pvgrub. Please note that in my case, I'm using hardened on Xen VPS instances where I have no access to dom0, so I can't answer your question about Xen & Xen-tools versions. I can tell you it's on Linode.

I'll spawn a testbed VPS instance and try the no-tls flag you suggest, as soon as possible.

*EDIT: Sorry, teh example was in the first post, by at_chaosLast edited by 229566 on Mon Aug 16, 2010 3:29 pm; edited 1 time in total

----------

## at_chaos

Hi guys, 

I updated the opening post with the dom0 from scratch howto. I added also some "assumptions" I run this setup at a datacenter, so it is headless (server profile), pure 64bit and hardened domUs. 

DomU Howto will follow in a few hours.

----------

## blueness

 *at_chaos wrote:*   

> Hi guys, 
> 
> I updated the opening post with the dom0 from scratch howto. I added also some "assumptions" I run this setup at a datacenter, so it is headless (server profile), pure 64bit and hardened domUs. 
> 
> DomU Howto will follow in a few hours.

 

Thanks, this is good stuff.  I'm going to try to reproduce all this and see how much hardening we can squeeze out before breaking stuff.

----------

## at_chaos

This is strange, if I enable all PaX features in Security Level Hardened Gentoo [Custom] the kernel works. If I choose the Security Level Hardened Gentoo [ server no rbac ] it is broken. The diff between this two configs show the following: 

```
diff .config-hardened-pax4 .config-hardened-pax5

4c4

< # Tue Aug 17 23:40:03 2010

---

> # Tue Aug 17 23:51:23 2010

352d351

< # CONFIG_EFI is not set

1988d1986

< # CONFIG_FUNCTION_TRACER is not set

1998d1995

< # CONFIG_STACK_TRACER is not set

2048c2045

< # CONFIG_GRKERNSEC_HARDENED_SERVER_NO_RBAC is not set

---

> CONFIG_GRKERNSEC_HARDENED_SERVER_NO_RBAC=y

2051c2048

< CONFIG_GRKERNSEC_CUSTOM=y

---

> # CONFIG_GRKERNSEC_CUSTOM is not set

2162a2160

> CONFIG_PAX_KERNEXEC=y

```

What about the last option "CONFIG_PAX_KERNEXEC=y". If I switch from server no rbac profile to custom this option seems to be not set. Is this a expected behaviour? I would expect that no matter what security level I chose before the options stay exactly the same when I switch to the custom sec level and all available PaX options are enabled. 

Beside that it seems that this CONFIG_PAX_KERNEXEC=y option breaks the startup.

Can somebody verify this please.

Working .config Sec. Level Custom  (all available PaX options enabled)

http://pastebin.ca/raw/1919262

Broken .config Sec. Level server no rbac 

http://pastebin.ca/raw/1919263

----------

## blueness

> CONFIG_PAX_KERNEXEC=y

This is progress.  KERNEXEC is the kernel land equivalent of PAGEEXEC which uses the NX bit to mark pages with the least possible privileges.  I'm not familiar with how the paravirt kernel does its work, but it would not surprise me if it tries to execute pages that it writes on the fly.

If this is the only problem, then I can easily add a Kconfig option to the [server] [server no rbac] etc which selects for [paravirt].

----------

## idella4

I shall add to this, once I get the system booting.

I took a gentoo system, converted it to selinux, one extra hardening layer.  I'm still tweaking the system to get the kernel to boot through all the selinux layers.

----------

## at_chaos

@idella4

Do you try to run the dom0 (host/hypervisor) with hardened profile or the domU? I tried to run xen with hardened setup back in 2007 and a few month ago as dom0 but I was not able to get it running at all. It would be great if we could have a running hardened dom0 with hardened domUs but for now I'm happy that we got hardened domUs  :Smile: 

----------

## idella4

I had started to update a vm anyway, and I had selected the selinux profile for the vm.

I'm working on getting it to boot.  I'm of the opinion the selinux side of it is holding it back.  The vm has some packages that won't re-emerge so I'm straightening them out.  The vm is booting from a generic ubuntu guest kernel, and it will not yet complete booting from the desired hardened kernel.

I'm not getting errors, the boot just stops.  It's all in the post.  It appears the selinux policy making is incomplete.

For some reason the ubuntu kernel gets past it.  It looks as if despite trying to turn off selinux on boot, it still examines it and finds some files not labeled.  At least I  can get it booted and in selinux.

Never touched selinux before so have to learn more again.  Looking forward to describing what it took.

I'm differing in not starting out with a hardened new system, rather converting a std one.

I'll get there.  I'd be happy establishing a hardened dom0 if it's warranted.

What have you got against the PaX option???

----------

## idella4

Right, here is my version of this.  This is a paralle howto to accompany the description for the gentoo hardened vm.

Scenario:  Using a standard x86 pc, use gentoo as the dom0 host, establish a gentoo 32 bit vm, 

profile of selinux [2007], 

booted by either a xen kernel selinux capable OR a gentoo-sources hardened kernel, paravirt, by use of pv-grub for the hardened kernel.

There are two gentoo hosts; one 32, one 64, interchangeable.

Disk /dev/sda:

/dev/sda hosts the gento hosts

/dev/sda10 a data partition, fs btrfs.

/dev/sda6 a karmic

/dev/sda8 hosts xen vms, a data partition.

The starting point for this, rather than create a new vm, an old gentoo vm is converted and updated to a hardened selinux profiled vm.

The source of the vm id a website I can't exactly remember the name, close to Zoos.org.  It hosts pre-made guest vms.

The vm used is a 2007 minimal guest.  The sense in using this for this exercise is that the selinux profile in portage is

```

gentoo64 linux-2.6-xen # eselect profile list

Available profile symlink targets:

  .......................................

  [8]   hardened/linux/amd64/10.0

  [9]   hardened/linux/amd64/10.0/no-multilib

  [10]  selinux/2007.0/amd64

  [11]  selinux/2007.0/amd64/hardened

```

[Replace amd64 with x86, same for both.

The inital vm is 2G in size.  Updating it quickly fills the space.  To alleviate the space burden, I established a second image file of 4G to house portage.

I soon still need to transfer the image of the vm to a new 5Gig image file, yielding

/mnt/images/gentoo-2007/gentoo-2007.img

/mnt/images/gentoo-2007/gentoo.swap

/mnt/images/gentoo-2007/store.img

/mnt/images/gentoo-2007/gentoo-se2007.img

The initial gentoo-2007.img can be discarded once the new gentoo-se2007.img is established.

Booting the newly created larger gentoo-se2007 was most interesting.  It required two separate guest kernels.

The vm had the new profile and a portage emerged.  An updated xen kernel missed  login due to a missing console device despite;

From the other post, the inittab is adjusted to 

```

add to SERIAL part

Quote:

h0:12345:respawn:/sbin/agetty 9600 hvc0 screen

```

```

# nano -w /etc/securetty

add hvc0 to the bottom

Quote:

hvc0

```

A guest kernel from ubuntu karmic managed to boot to a rescue console.  

From there, the bulk of the conversion was put in place.

Once the selinux content and the guest harened kernel were prepared, it booted into an selinux state. 

............................................................................................

XEN packages & kernels;

As above, I utilised the karmic prepared guest kernel for initial booting.

I had also emerged gentoo xen kernel, and the xensource kernel.

In the gentoo32, it was fully updated, the xen package xen-4.0.0.

To execute this, the gentoo sourced xen and xen kernel were put aside.

Updating of udev caused a corruption of the making of vif devices in a xen environment.

A bug was submitted, which lead to acquiring a patch for the xen kernel.

The patch was of xensource origin. It applied effectively only to the xensource kernel, 

which is substantially larger than the gentoo xen kernel.

Curiously, the upgraded version of the xensource kernel [xen-2.6.32-19] was effective in overcoming udev-160.

The patched 2.6.31.13 faultered just like the gentoo kernels.

Alternately, the gentoo64 host has packages of prior versions, i.e. not up to date.

Those xen packages, xen-3.4.3 and udev-150 or so, all work effectively.

This difference aside, the gentoo hosts are interchangeable in hosting the gentoo guest.

Installing Xen and Xen kernel

To acquire the xensource kernel.

[/code]

 git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen

cd linux-2.6-xen

git checkout origin/xen/master -b xen/master

git pull

For a gentoo kernel, the usual emerge xen-sources.

dom0 Hypervisor

```

gentoo64 linux-2.6-xen # uname -a                                         

Linux gentoo64 2.6.34-xen-amd64 #6 SMP Mon Aug 2 16:04:32 Local time zone must be set--see zic m x86_64 Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz GenuineIntel GNU/Linux      [64 host]                                                                          

gentoo64 linux-2.6-xen # ls /boot 

config-2.6.34-xen-gentoo-amd64

initrd.img-2.6.34-xen-gentoo-amd64

kernel-2.6.34-xen-gentoo-amd64

xen-3.4.2.gz

xen-3.4.3-rc6-pre.gz

xen-syms-3.4.2

xen-syms-3.4.3-rc6-pre

```

Set xen related useflags 

/etc/make.conf

```

gentoo64 linux-2.6-xen # cat /mnt/genny/etc/make.conf                              

# These settings were set by the catalyst build script that automatically built this stage                                                                            

# Please consult /etc/make.conf.example for a more detailed example                

CFLAGS="-march=core2 -fomit-frame-pointer -pipe -O2 -mno-tls-direct-seg-refs -ggdb"

CHOST="i686-pc-linux-gnu"                                                          

CXXFLAGS="${CFLAGS}"                                                               

MAKEOPTS="-j2" 

DISTDIR="/mnt/gentoo/distfiles"

FEATURES="${FEATURES} multilib-strict parallel-fetch"

VIDEO_CARDS="fbdev nvidia vesa v4l"

INPUT_DEVICES="evdev"

ACCEPT_KEYWORDS="~x86"

ACCEPT_LICENSE="dlj-1.1"

QEMU_SOFTMMU_TARGETS="arm cris i386 m68k microblaze mips mips64 mips64el mipsel ppc ppc64 ppcemb sh4 sh4eb sparc sparc64 x86_64" QEMU_USER_TARGETS="alpha arm armeb cris i386 m68k microblaze mips mipsel ppc ppc64 ppc64abi32 sh4 sh4eb sparc sparc32plus sparc64 x86_64"

PORTDIR="/usr/portage"

```

Configs.

This is covered well enough by the other post.  No need to repeat the content/

The karmiv domU config is in fact here.

The XenParavirtOps is outlined at xensource

.........................................................................

Converting the system

Updating the vm from that period is not so hard, using a profile of selinux/2007.0 makes it easier.

In brief, the initial emerge --sync creates a portage block which took me a while to break.

Once found, it's standard updating.  The initial update need be to an intermediate portage version.

Update a few key pacakges such as glibc, gtk, gcc itself.  Initially, emerge te intermediate version of portage with the -O option.

Then eselect the profile, set number 11, and then begin updating and converting.

To begin, emerge points you towards gcc and glibc and python.  

NOTE: at this time, it's required to mask glibc-2.12.1, then emerge will select to update to the preferred 2.11.2

gcc first, then you must gcc-config to the newly emerged gcc-4.4.4 or 4.4.6 so glibc will l compile.

Then, there is the gentoo selinux guide

with other gentoo selinux support docs to guide the conversion.

Once gcc and glibc are in place, then just follow the cited selinux guide in selecting and new and 

re-emerging packages to convert the sytem to selinux mode.

Do these before attempting to update the system or world.

In the guide, Bringing the System up to Date cites a required method to bypass a block 

re e2fsprogs which includes a world update.  python-updater tends to attempt to emerge non existant package; just emerge those that are there manually.

```

192 ~ # emerge -uDN world --jobs=5 --load-average=4.4 && revdep-rebuild

 * IMPORTANT: 4 news items need reading for repository 'gentoo'.

 * Use eselect news to read news items.

Calculating dependencies... done!

>>> Verifying ebuild manifests

>>> Starting parallel fetch

>>> Emerging (1 of 159) sys-libs/zlib-1.2.5-r2

```

emerge the hardened-sources early in the process.  Some of the packages call on the content of a kernel to complete.

Be prepared to insert the odd sym-link to staisfy the configure states of some packages.  

e.g. falloc.h. present in the kernel.  Also, the twice I've done this, the linking of binutils is broken.

To get the compiler back you need to symlink all binutils executables to /bin/

grub

To utilise the hardened kernel, grub is required.  The image file need be not sub-partitioned.  On emergeing grub, it's enough to

```

emerge --configure grub

```

just nominating the /boot folder to install.  PV-grub will then find the kernel.[/quote]

Booting.

Initially, the booting of the guest was done via;

```

#

# Configuration file for the Xen instance lenny01, created

# by xen-tools 4.1 on Sun May 16 01:10:35 2010.

#

#  Hostname

name        = 'gentoo-2008'

#

#  Kernel + memory size

#

#kernel      = '/mnt/genny/boot/kernel-2.6.32.13-xen-SE'

#ramdisk     = '/mnt/genny/boot/initrd.img-2.6.32.13-xen-SE'

kernel       = '/mnt/ubuntu//boot/vmlinuz-2.6.31.6-xenU'

ramdisk      = '/mnt/ubuntu/boot/initramfs.img-2.6.31.6-xenU'

memory      = '550'

#

#  Disk device(s).

#

root        = '/dev/xvda2 ro console=tty0 enforcing=0'

disk        = [

                  'file:/mnt/images/images/gentoo-2007/gentoo-2008-0.img,xvda2,w',

                  'file:/mnt/images/images/gentoo-2007/gentoo.swap,xvda1,w',

                  'file:/mnt/ubuntu/store/store.img,xvdb,w',

#                  'phy:/dev/sda10,xvdc,w'

             ]

#

#  Physical volumes

#  Networking

#

dhcp        = 'dhcp'

vif         = [ 'mac=00:16:3E:59:C4:6E,bridge=eth0' ]

vif         = [ ' ' ]

#

#  Behaviour

#

on_poweroff = 'destroy'

on_reboot   = 'restart'

on_crash    = 'restart'

#vfb=['type=vnc,vncunused=1']

extra = '4 console=hvc0'

```

Note the two kernels.  The kernel not commented is the karmic guest kernel.

The other kernel is the xensource kernel.

The xensource kernel can provide the selinux config for the hardened gentoo guest.

 *Quote:*   

> 
> 
>  .config - Linux Kernel v2.6.32.19 Configuration
> 
>  ─────────────────────────────────────────────────────────────────────────────────
> ...

 

Once built to a required level, the xen kernel can boot the guest in hardened mode.

Alternatively, the hardened kernel can boot the guest.  The file that boots the domU, gentoo8.pv-grub

```

----------------------------------------------------------------------------

# PV GRUB image file.

kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"

# Optional provided menu.lst.

#ramdisk = "/boot/grub/grub.conf"

# Sets path to menu.lst

extra = "(hd1)/boot/grub/menu.lst"

# can be a TFTP-served path (DHCP will automatically be run)

# extra = "(nd)/netboot/menu.lst"

# can be configured automatically by GRUB's DHCP option 150 (see grub manual)

extra = "4 console=hvc0"

# Initial memory allocation (in megabytes) for the new domain.

#

# WARNING: Creating a domain with insufficient memory may cause out of

#          memory errors. The domain needs enough memory to boot kernel

#          and modules. Allocating less than 32MBs is not recommended.

memory = 256

# A name for your domain. All domains must have different names.

name = "gentoo-2007"

# 128-bit UUID for the domain.  The default behavior is to generate a new UUID

# on each call to 'xm create'.

uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"

vcpus = 2

#

dhcp        = 'dhcp'

vif         = [ 'mac=00:16:3E:59:C4:6E,bridge=eth0' ]

disk        = [

                  'file:/mnt/images/images/gentoo-2007/gentoo-2008-0.img,xvda2,w',

                  'file:/mnt/images/images/gentoo-2007/gentoo.swap,xvda1,w',

                  'file:/mnt/karmic64/store/store.img,xvdb,w',

#                  'phy:/dev/sda10,xvdc,w'

             ]

#

#vfb = [ 'vnc=1,vnclisten=0.0.0.0,vncunused=1' ]

#

extra = '4 console=hvc0'

#

on_poweroff = 'destroy'

on_reboot   = 'restart'

on_crash    = 'restart'

```

PV-grub comes from the xensource package, compiled in gentoo.  Like pygrub, it boots the resident kernel.

I need not go into networking setup;  it's standard gentoo and is outlined in the other post.

selinux can be temperamental.  On changing kernels, extensive relabeling was required.

In parvavirt booting the hardened kernel

```

root@gentoo_pristine:/home/idellagentoo_pristine idella # uname -a

Linux gentoo_pristine 2.6.34-hardened-r2 #2 SMP Fri Aug 27 13:00:32 WST 2010 i686 Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz GenuineIntel GNU/Linux

root@gentoo_pristine:/home/idellagentoo_pristine idella # hostname

gentoo_pristine

root@gentoo_pristine:/home/idellagentoo_pristine idella # cat /selinux/enforce

1

root@gentoo_pristine:/home/idellagentoo_pristine idella # sestatus

SELinux status:                 enabled

SELinuxfs mount:                /selinux

Current mode:                   enforcing

Mode from config file:          enforcing

Policy version:                 24

Policy from config file:        targeted

```

----------

## 229566

 *at_chaos wrote:*   

> 
> 
> Beside that it seems that this CONFIG_PAX_KERNEXEC=y option breaks the startup.
> 
> Can somebody verify this please.
> ...

 

I can confirm that I can boot into PAX-enabled kernel if I choose custom instead of server profile, ie. without CONFIG_PAX_KERNEXEC.

----------

## ygeorgiev

New xen: http://lists.xensource.com/archives/html/xen-devel/2010-08/msg01526.html

 *Quote:*   

> Xen 4.0.1 changes
> 
>     * Many bugfixes. Upgrading is recommended for all Xen 4.0.0 users.
> 
>     * Default pvops kernel is now Linux 2.6.32.x.
> ...

 

----------

## idella4

yes, well, the xen in this gentoo 32 has been xen-4.0.1 for a while, from xensource.

gentoo's xen-4.0.0 doesn't work, posted re this s few weeks ago.

----------

## blueness

 *Ratrace wrote:*   

>  *at_chaos wrote:*   
> 
> Beside that it seems that this CONFIG_PAX_KERNEXEC=y option breaks the startup.
> 
> Can somebody verify this please.
> ...

 

Of all the server profiles, is it just CONFIG_PAX_KERNEXEC that is causing the problem with a xen paravirt guest?  I can confirm that with a xen full virt guest the GRSEC/PaX settings do not seem to make a difference.

I'm considering creating other preset profiles, but the issue is somewhat complex.  For example, with KVM its the host that appears to be the problem.  There you need to set KERNEXEC=n UDEREF=n while the client can have pretty much anything, even if it is using virtio instead of emulated hardware. (See https://bugs.gentoo.org/show_bug.cgi?id=328623).

----------

## newtonian

 *Quote:*   

> #####todo check hw-clock error, minor problem 

 

This should fix your hw-clock error:

add xenfs to /etc/fstab:

```

xenfs        /proc/xen    xenfs    defaults           0 0

```

source: https://bugs.gentoo.org/show_bug.cgi?id=96240

----------

## dummys

Did anyone has his Xen domU Gentoo hardened with 3.4.5 kernel and the NX bit enable ?

I try several things and can't get the NX Bit enable at all...

On the same XenServer, i installed an Centos box and when i cat /cpu/procinfo the nx bit is here.

Anyone has an idea ?

PS : sorry for my bad english

----------

