# How to get cgroup, vserver and host rules rules running

## khippy

Lately I stumbled about the 2.6 cgroup kernel feature which is able to rule device management based on user/group/process, e. g. to divide count of cpu cores to different groups. Setting up kernel parameters for cgroup was easy and after rebooting the pc I was surprised about the fact that my vserver was using cgroup out of the box (new to cgroup I had set _all_ possible kernel configs for cgroup, so CONFIG_CGROUP_NS=y, too). Reducing the cpu cores for the vserver was easy, too. Lucky me was then happily installing dev-libs/libcgroup from the sunrise overlay.

Note: Vserver uses /dev/cgroup as mountpoint!

Getting dev-libs/libcgroup running together with vserver was a severity and that's why I want to share because I was succesful in the end.

dev-libs/libcgroup installes config files at /etc/cgroup/cgconfig.conf, /etc/cgroup/cgrules.conf, /etc/conf.d/cgconfig and /etc/conf.d/cgred.

/etc/init.d/cgconfig parses /etc/cgroup/cgconfig.conf and creates the <mountpoint>/cgroup/<files>, /etc/init.d/cgred then starts cgrulesengd which parses /etc/cgroup/cgrules.conf to move tasks from <mountpoint>/cgroup/tasks to <mountpoint>/cgroup/<usr/group directory>/tasks. The latter is what makes the left over cpu cores being reserveable for host system as I wanted which is what didn't work out of the box (which doesn't aplly to some system processes, those PIDs can't be moved deeper into the /dev/cgroup/ structure, thus they use all cores.).

Yes, I have got different cores for the host user processes, it is not restricted to xterm or the like, so read on.

Setting /etc/cgroup/* to my needs from the documentation I had found seemed to work at the prompt after some tests, but rebooting with them didn't work and the reason is that I have a vserver running. Vserver places the mountpoint of cgroup to /dev/cgroup and that does not work out of the box with dev-libs/libcgroup. Furthermore, standard procedure is /etc/init.d/cgconfig and /etc/conf.d/cgred are executed _before_ /etc/conf.d/util-vserver and /etc/init.d/vserver.*. Following this I managed to create the /dev/cgroup directory by modifying /etc/init.d/fsck which is running early at the boot process, thus /etc/init.d/cgconfig was working, though the vserver wasn't able to use that directory anymore. Lateron I got it working, but that was a mess, too. I noticed the vserver PIDs the problem which made me wishing for a namespace target at cgrules.conf. In the end, as I didn't want to change vserver files, I modified the order of libcgroup at the boot process to run as very last which solved almost all problems.

/etc/init.d/cgconfig:

```

61c61,62

<       need localmount

---

>       need local

>

```

/etc/init.d/cgred:

```
14c14

<       need cgconfig localmount

---

>       need cgconfig local

```

Now /etc/init.d/cgconfig is executed at the very end after /etc/init.d/local and also after /etc/init.d/vserver. See my

/etc/cgroup/cgconfig.conf:

```
group foo {

        perm {

                task {

                        uid = foo;

                        gid = foo;

                }

                admin {

                        uid = root;

                        gid = root;

                }

        }

        cpu {

                cpuset.mems = 0;

                cpuset.cpus = 0-1;

        }

}

group bar {

        perm {

                task {

                        uid = bar;

                        gid = bar;

                }

                admin {

                        uid = root;

                        gid = root;

                }

        }

        cpu {

                cpuset.mems = 0;

                cpuset.cpus = 0-1;

        }

}

group root {

        perm {

                task {

                        uid = root;

                        gid = root;

                }

                admin {

                        uid = root;

                        gid = root;

                }

        }

        cpu {

                cpuset.mems = 0;

                cpuset.cpus = 0-1;

        }

}

```

Compare with vserver config at /etc/vservers/<vservername>/cgroup/cpuset.mems and /etc/vservers/<vservername>/cgroup/cpuset.cpus. The first reads 0 and the latter 2-3.

My aim was to devide the cpu cores, that is changing cpuset.cpus. To be able to alter e. g. /dev/cgroup/foo/cpuset.cpus, /dev/cgroup/foo/cpuset.mems needs to get a value because it is empty - my x86 pc has 0 at /dev/cgroup/cpuset.mems.

There was a problem with cgrulesengd or runscript, cgrulesengd doesn't produce a pid, little to change to get one:

/etc/init.d/cgred:

```
25,26c25,26

<       start-stop-daemon --start --exec "${CGRULESENGD}" \

<               --pidfile "${PID_FILE}" -- ${options} >/dev/null

---

>       start-stop-daemon --start --pidfile "${PID_FILE}" --make-pidfile \

>               --background --exec "${CGRULESENGD}" -- ${options} >/dev/null

---

>       start-stop-daemon --start --pidfile "${PID_FILE}" --make-pidfile \

>               --background --exec "${CGRULESENGD}" -- ${options} >/dev/null

32c32

<       start-stop-daemon --stop --exec "${CGRULESENGD}" --pidfile "${PID_FILE}"

---

>       start-stop-daemon --stop --pidfile "${PID_FILE}" --exec "${CGRULESENGD}"

39c39

<               --exec "${CGRULESENGD}" --pidfile "${PID_FILE}"

---

>               --background --pidfile "${PID_FILE}" --make-pidfile --exec "${CGRULESENGD}"

```

That works if you modify

/etc/conf.d/cgred:

```
7,8c7,8

< NODAEMON=""

< #NODAEMON="--nodaemon"

---

> #NODAEMON=""

> NODAEMON="--nodaemon"

12,13c12,13

< LOG=""

< #LOG="--nolog"

---

> #LOG=""

> LOG="--nolog"

17c17

< #PID_FILE=/var/run/cgred.pid

---

> PID_FILE=/var/run/cgred.pid

```

This lets cgrulesengd work at foreground, runscript backgrounds itself then by the --background option and then --make-pidfile works perfectly.

See my

/etc/cgroup/cgred:

```
foo          *                  foo/

bar        *                    bar/

root            *               root/

```

You can check /dev/cgroup/foo/tasks' content to verify cgrulesengd working fine, or by killing its process and start as root

```
cgrulesengd -n

```

at the foreground. You should see OK messages, else you have a problem.

A note about /etc/init.d/cgred: It says PID_FILE=${PIF_FILE:-"/var/run/cgred.pid"} at line 11. I don't understand that little bash or runscript magic, but I suppose it should read PID_FILE=${PID_FILE:-"/var/run/cgred.pid"} (PID_FILE instead of PIF_FILE), though it doesn't make a difference to me.

Another note about changing /etc/init.d/ files. I encountered boot problems by doing so, maybe because of baselayout-2 or openrc. So I advise you strongly to execute

```
rc-update -u

```

after having finished modifying your scripts at /etc/init.d/ and before rebooting!

I hope this experience report is useful or interesting to you.

See my cpu.conky here: http://img818.imageshack.us/img818/1111/cgroupcpus.png

Happy cgrouping!

----------

