# Hardening PPC64 w/32-bit userland REMOVED 32-bit glibc!

## jhardin

Folks:

I'm installing a new PowerMac G5 for use as a firewall and server, and I am recovering from a rather unpleasant experience related to trying to harden it...

I'm using 64-bit kernel and 32-bit userland loaded off the current install CD and stage3. It was all working properly with the default profile. I downloaded the hardened kernel sources, configured the kernel per http://www.gentoo.org/proj/en/hardened/pax-quickstart.xml , built, installed and rebooted and it all worked fine.

At this point the next step is to harden userland.

I switched my profile to hardened ppc64 w/32-bit userland:

```
# eselect profile show

Current /etc/make.profile symlink:

  /usr/portage/profiles/hardened/linux/powerpc/ppc64/32bit-userland
```

I then emerged binutils, gcc and glibc (rather than virtual/libc).

binutils and gcc completed successfully, and the glibc build churned along promisingly, but then when it completed and installed, my system was rendered unusable. It seems that the above profile wants to build a 64-bit userland, at least as far as glibc is concerned:

```
# cat /var/tmp/portage/sys-libs/glibc-2.12.2/temp/build.log

 * Package:    sys-libs/glibc-2.12.2

 * Repository: gentoo

 * Maintainer: toolchain@gentoo.org

 * USE:        elibc_glibc hardened kernel_linux nls ppc64 userland_GNU

make -j3 -s glibc-test 

make -j3 -s glibc-test 

>>> Unpacking source...
```

Wait, what? USE ppc64? Is that correct for a 32-bit userland?

```
>>> Source unpacked in /var/tmp/portage/sys-libs/glibc-2.12.2/work

>>> Compiling source in /var/tmp/portage/sys-libs/glibc-2.12.2/work/glibc-2.12.2 ...

 *             ABI:   ppc64

 *          CBUILD:   powerpc-unknown-linux-gnu

 *           CHOST:   powerpc-unknown-linux-gnu

 *         CTARGET:   powerpc-unknown-linux-gnu

 *      CBUILD_OPT:   

 *     CTARGET_OPT:   

 *              CC:   powerpc-unknown-linux-gnu-gcc 

 *          CFLAGS:   -pipe -freorder-blocks -O2 -fno-strict-aliasing -fno-stack-protector

 *       Manual CC:   powerpc-unknown-linux-gnu-gcc 
```

"-fno-stack-protector"? Isn't this supposed to be hardened?

```
 * Configuring GLIBC for nptl

/var/tmp/portage/sys-libs/glibc-2.12.2/work/glibc-2.12.2/configure --enable-stackguard-randomization --enable-old-ssp-compat --enable-add-ons=nptl,c_stubs,libidn,ports --enable-kernel=2.6.9 --without-selinux --without-cvs --enable-bind-now --build=powerpc-unknown-linux-gnu --host=powerpc-unknown-linux-gnu --disable-profile --without-gd --with-headers=/usr/include --prefix=/usr --libdir=/usr/lib64 --mandir=/usr/share/man --infodir=/usr/share/info --libexecdir=/usr/lib64/misc/glibc --disable-multi-arch

checking build system type... powerpc-unknown-linux-gnu

checking host system type... powerpc-unknown-linux-gnu

checking for powerpc-unknown-linux-gnu-gcc... powerpc-unknown-linux-gnu-gcc 

...

strip: powerpc-unknown-linux-gnu-strip --strip-unneeded -R .comment

   usr/lib64/crtn.o

   usr/lib64/libieee.a

   usr/lib64/libc_stubs.a

...

   lib64/libresolv-2.12.2.so

   lib64/libpcprofile.so

   lib64/libnss_hesiod-2.12.2.so

   lib64/ld-2.12.2.so

   lib64/libnss_nis-2.12.2.so

   lib64/libcrypt-2.12.2.so

   lib64/libnss_dns-2.12.2.so

   lib64/libc-2.12.2.so

   lib64/libthread_db-1.0.so

   lib64/libnss_nisplus-2.12.2.so

   lib64/libmemusage.so

   lib64/libSegFault.so

   lib64/libnss_files-2.12.2.so

   lib64/libpthread-2.12.2.so

   lib64/libdl-2.12.2.so

   lib64/librt-2.12.2.so

   lib64/libcidn-2.12.2.so

   lib64/libnsl-2.12.2.so

   lib64/libutil-2.12.2.so

   lib64/libm-2.12.2.so

   lib64/libBrokenLocale-2.12.2.so

   lib64/libnss_compat-2.12.2.so

   lib64/libanl-2.12.2.so

...

>>> Completed installing glibc-2.12.2 into /var/tmp/portage/sys-libs/glibc-2.12.2/image/

ecompressdir: bzip2 -9 /usr/share/info

ecompressdir: bzip2 -9 /usr/share/man

making executable: usr/lib64/libc.so

making executable: usr/lib64/libpthread.so

...

 * >>> SetUID: [chmod go-r] /usr/lib64/misc/glibc/pt_chown ...

 [ ok ]

```

...at this point the whole system fell apart, since /lib/libc.so.6 and related files have been deleted...

```
[Errno 2] No such file or directory:

   /bin/bash -c "/usr/lib/portage/bin/ebuild.sh" postrm

Traceback (most recent call last):

  File "/usr/lib/portage/pym/portage/process.py", line 250, in spawn

    env, gid, groups, uid, umask, pre_exec)

  File "/usr/lib/portage/pym/portage/process.py", line 379, in _exec

    os.execve(binary, myargs, env)

  File "/usr/lib/portage/pym/portage/__init__.py", line 222, in __call__

    rval = self._func(*wrapped_args, **wrapped_kwargs)

OSError: [Errno 2] No such file or directory

 * The ebuild phase 'postrm' has exited unexpectedly. This type of behavior

 * is known to be triggered by things such as failed variable assignments

 * (bug #190128) or bad substitution errors (bug #200313). Normally, before

 * exiting, bash should have displayed an error message above. If bash did

 * not produce an error message above, it's possible that the ebuild has

 * called `exit` when it should have called `die` instead. This behavior

 * may also be triggered by a corrupt bash binary or a hardware problem

 * such as memory or cpu malfunction. If the problem is not reproducible or

 * it appears to occur randomly, then it is likely to be triggered by a

 * hardware problem. If you suspect a hardware problem then you should try

 * some basic hardware diagnostics such as memtest. Please do not report

 * this as a bug unless it is consistently reproducible and you are sure

 * that your bash binary and hardware are functioning properly.

```

My logged-in shells continued to work, to the extent that I could "cd" and "echo *" as a way to look at the filesystem, but that's about it. Nothing that depended on libc being there (like fork+exec a new process) worked.

I'm recovering the 32-bit glibc libraries off the install CD now, and I will try to switch back to the working non-hardened 32-bit userland profile and rebuild it to see if I can clean this up. Unfortunately none of the options in "eselect profile list" look promising:

```
# eselect profile list

Available profile symlink targets:

  [1]   default/linux/powerpc/ppc64/10.0/64bit-userland

  [2]   default/linux/powerpc/ppc64/10.0/64bit-userland/desktop

  [3]   default/linux/powerpc/ppc64/10.0/64bit-userland/desktop/gnome

  [4]   default/linux/powerpc/ppc64/10.0/64bit-userland/desktop/kde

  [5]   default/linux/powerpc/ppc64/10.0/64bit-userland/developer

  [6]   default/linux/powerpc/ppc64/10.0/64bit-userland/server

  [7]   hardened/linux/powerpc/ppc64/64bit-userland
```

So, question: is the hardened ppc64 32bit-userland profile screwed up? Or did I do something incorrect, like picking the wrong profile?

----------

## Halcy0n

It looks like that profile is currently broken from my quick check.  I will look at fixing it later when I am in front of my dev box.

----------

## jhardin

 *Halcy0n wrote:*   

> It looks like that profile is currently broken from my quick check.  I will look at fixing it later when I am in front of my dev box.

 

Thanks.

Can you confirm whether or not SSP and PIC/PIE are even available on ppc32? I have "hardened" and "pic" in make.conf USE, and I've rebuilt the toolchain, but I see this against a program that I just merged:

```
~ # bash checksec.sh --file /usr/sbin/ntpd

RELRO           STACK CANARY      NX            PIE                     FILE

Partial RELRO   No canary found   NX enabled    No PIE                  /usr/sbin/ntpd
```

gcc-config -l doesn't suggest the hardening options are available/controllable any more - they were offered at one point (when I had the hardened profile selected?):

```
~ # gcc-config -l

 [1] powerpc-unknown-linux-gnu-4.4.5 *

 [2] powerpc64-unknown-linux-gnu-4.1.2 *
```

My current profile:

```
~ # eselect profile list

Available profile symlink targets:

  [1]   default/linux/powerpc/ppc32/10.0

  [2]   default/linux/powerpc/ppc32/10.0/desktop

  [3]   default/linux/powerpc/ppc32/10.0/desktop/gnome

  [4]   default/linux/powerpc/ppc32/10.0/desktop/kde

  [5]   default/linux/powerpc/ppc32/10.0/developer

  [6]   default/linux/powerpc/ppc32/10.0/server

  [7]   default/linux/powerpc/ppc64/10.0/32bit-userland *

  [8]   default/linux/powerpc/ppc64/10.0/32bit-userland/desktop

  [9]   default/linux/powerpc/ppc64/10.0/32bit-userland/desktop/gnome

  [10]  default/linux/powerpc/ppc64/10.0/32bit-userland/desktop/kde

  [11]  default/linux/powerpc/ppc64/10.0/32bit-userland/developer

  [12]  default/linux/powerpc/ppc64/10.0/32bit-userland/server

  [13]  hardened/linux/powerpc/ppc32

  [14]  hardened/linux/powerpc/ppc64/32bit-userland

~ # eselect profile show

Current /etc/make.profile symlink:

  default/linux/powerpc/ppc64/10.0/32bit-userland

```

And the current toolchain USEs:

```
~ # equery u binutils gcc glibc

[ Legend : U - final flag setting for installation]

[        : I - package is installed with flag     ]

[ Colors : set, unset                             ]

 * Found these USE flags for sys-devel/binutils-2.20.1-r1:

 U I

 - - multislot   : Allow for multiple versions of binutils to be emerged at once for same CTARGET

 - - multitarget : Adds support to binutils for cross compiling (does not work with gas)

 + + nls         : Adds Native Language Support (using gettext - GNU locale utilities)

 - - test        : Workaround to pull in packages needed to run with FEATURES=test. Portage-2.1.2 handles this internally, so don't set it in

                   make.conf/package.use anymore

 - - vanilla     : Do not add extra patches which change default behaviour; DO NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning changes

                   drastically

 * Found these USE flags for sys-devel/gcc-4.4.5:

 U I

 + + altivec   : Adds support for optimizations for G4 and G5/ppc970 processors

 - - bootstrap : !!internal use only!! DO NOT SET THIS FLAG YOURSELF!, used during original system bootstrapping [make stage2]

 - - build     : !!internal use only!! DO NOT SET THIS FLAG YOURSELF!, used for creating build images and the first half of bootstrapping [make

                 stage1]

 - - doc       : Adds extra documentation (API, Javadoc, etc)

 - - fortran   : Adds support for fortran (formerly f77)

 - - gcj       : Enable building with gcj (The GNU Compiler for the Javatm Programming Language)

 - - graphite  : Add support for the framework for loop optimizations based on a polyhedral intermediate representation

 - - gtk       : Adds support for x11-libs/gtk+ (The GIMP Toolkit)

 + + mudflap   : Add support for mudflap, a pointer use checking library

 - - multislot : Allow for SLOTs to include minor version (3.3.4 instead of just 3.3)

 + + nls       : Adds Native Language Support (using gettext - GNU locale utilities)

 - - nocxx     : Disable support for C++ (DON'T USE THIS UNLESS YOU KNOW WHAT YOU'RE DOING)

 - - nopie     : Disable PIE support (NOT FOR GENERAL USE)

 - - nossp     : Disable SSP support (NOT FOR GENERAL USE)

 + + nptl      : Enable support for Native POSIX Threads Library, the new threading module (requires linux-2.6 or better usually)

 - - objc      : Build support for the Objective C code language

 - - objc++    : Build support for the Objective C++ language

 - - objc-gc   : Build support for the Objective C code language Garbage Collector

 + + openmp    : Build support for the OpenMP (support parallel computing), requires >=sys-devel/gcc-4.2 built with USE="openmp"

 - - test      : Workaround to pull in packages needed to run with FEATURES=test. Portage-2.1.2 handles this internally, so don't set it in

                 make.conf/package.use anymore

 - - vanilla   : Do not add extra patches which change default behaviour; DO NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning changes

                 drastically

 * Found these USE flags for sys-libs/glibc-2.12.2:

 U I

 - - debug        : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see

                    http://www.gentoo.org/proj/en/qa/backtraces.xml

 - - gd           : Adds support for media-libs/gd (to generate graphics on the fly)

 - - glibc-omitfp : Configure glibc with --enable-omitfp which lets the build system determine when it is safe to use -fomit-frame-pointer

 + + nls          : Adds Native Language Support (using gettext - GNU locale utilities)

 - - profile      : Adds support for software performance analysis (will likely vary from ebuild to ebuild)

 - - vanilla      : Do not add extra patches which change default behaviour; DO NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning changes

                    drastically

```

(..."openmp"? I'll have to disable that...)

I see -nossp, but since that's disabled shouldn't SSP be automatically a part of new compiles? Likewise -nopie?

----------

## blueness

I know you're working with hardened/linux/powerpc/ppc64/32bit-userland, but for the record this is what I have on hardened/linux/powerpc/ppc32:

```
gentoo-ppc ~ # ./checksec.sh --proc-all

         COMMAND    PID RELRO             STACK CANARY           NX            PIE                     ASLR

            init      1 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

            sshd  29873 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

            bash  29875 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          pickup  30539 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

           udevd   7106 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

       syslog-ng   8754 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

       syslog-ng   8755 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

             atd   8816 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

           fcron   8883 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          dhcpcd   9323 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          master   9536 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

            qmgr   9549 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

            sshd   9598 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

           login   9730 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          agetty   9731 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          agetty   9732 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          agetty   9733 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          agetty   9734 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

          agetty   9735 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

            bash   9736 Full RELRO        Canary found           NX enabled    PIE enabled             ASLR enabled         

```

```
gentoo-ppc ~ # gcc-config -l

 [1] powerpc-unknown-linux-gnu-4.3.4

 [2] powerpc-unknown-linux-gnu-4.3.4-vanilla

 [3] powerpc-unknown-linux-gnu-4.4.5 *

 [4] powerpc-unknown-linux-gnu-4.4.5-hardenednopie

 [5] powerpc-unknown-linux-gnu-4.4.5-hardenednopiessp

 [6] powerpc-unknown-linux-gnu-4.4.5-hardenednossp

 [7] powerpc-unknown-linux-gnu-4.4.5-vanilla

```

```
gentoo-ppc ~ # equery u binutils gcc glibc 

[ Searching for packages matching glibc... ]

[ Colour Code : set unset ]

[ Legend : Left column  (U) - USE flags from make.conf              ]

[        : Right column (I) - USE flags packages was installed with ]

[ Found these USE variables for sys-libs/glibc-2.11.3 ]

 U I

 - - crosscompile_opts_headers-only : Build the toolchain against cross-headers only.

 - - debug                          : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml

 - - gd                             : Adds support for media-libs/gd (to generate graphics on the fly)

 - - glibc-omitfp                   : Configure glibc with --enable-omitfp which lets the build system determine when it is safe to use -fomit-frame-pointer

 + + hardened                       : activate default security enhancements for toolchain (gcc, glibc, binutils)

 - - multilib                       : On 64bit systems, if you want to be able to compile 32bit and 64bit binaries

 + + nls                            : Adds Native Language Support (using gettext - GNU locale utilities)

 - - profile                        : Adds support for software performance analysis (will likely vary from ebuild to ebuild)

 - - selinux                        : !!internal use only!! Security Enhanced Linux support, this must be set by the selinux profile or breakage will occur

 - - vanilla                        : Do not add extra patches which change default behaviour; DO NOT USE THIS ON A GLOBAL SCALE as the severity of the meaning changes drastically

```

I see nothing wrong with the above.  My ppc64 is currently on the fritz.

----------

## jhardin

 *blueness wrote:*   

> I know you're working with hardened/linux/powerpc/ppc64/32bit-userland, but for the record this is what I have on hardened/linux/powerpc/ppc32:
> 
> ```
> gentoo-ppc ~ # ./checksec.sh --proc-all
> 
> ...

 

Envy! Thanks for showing it is possible...

 *Quote:*   

> 
> 
> ```
> gentoo-ppc ~ # gcc-config -l
> 
> ...

 

At one point my gcc-config listed those options. It does not any longer. I suppose that's because I'm not using the hardened profile at the moment.

 *Quote:*   

> 
> 
> ```
> gentoo-ppc ~ # equery u binutils gcc glibc 
> 
> ...

 

My glibc doesn't list the hardened USE at all.

I suspect I'll have to wait until the hardened/ppc64/32bit-userland profile is fixed.

----------

## jhardin

 *Halcy0n wrote:*   

> It looks like that profile is currently broken from my quick check.  I will look at fixing it later when I am in front of my dev box.

 

Is the hardened/linux/powerpc/ppc64/32bit-userland profile fixed now? I would really like to update my slotted cross gcc from 4.1.2 to 4.5.3 so I can build the 3.x-series hardened kernels without warnings, and this is now a production server so I don't want to have to recover the 32-bit stuff off the install CD again if the profile is still b0rked...

Alternatively, if I could get directions to just upgrade the slotted cross gcc without changing my profile I'd appreciate it. Using crossdev doesn't seem to work. Would it work to select the ppc64/32bitUL profile just long enough to compile binutils and gcc, and then switch the profile back?

```
athena ~ # gcc-config -l

 [1] powerpc-unknown-linux-gnu-4.4.5

 [2] powerpc-unknown-linux-gnu-4.5.3 *

 [3] powerpc64-unknown-linux-gnu-4.1.2 *
```

----------

