# symlinks in custom initramfs not resolved at boot [solved]

## triquetra

I have been trying to create an initramfs with genkernel-next, but without success, as documented here.

In an attempt to troubleshoot the issue, I tried to create an initramfs with dracut, and even tried multiple permutations of a custom initramfs.  None of them worked.

I have a booting system right now only because I am booting with an initramfs created with the old genkernel, but it seems I am unable to create a new functioning initramfs by any means.

I have banged my head against this problem for many hours over several days, and I am at a loss to figure out what I may be doing wrong.  Any ideas?Last edited by triquetra on Mon Feb 23, 2015 7:26 pm; edited 2 times in total

----------

## NeddySeagoon

triquetra,

Luckily systemd has nothing to do with an initramfs, or I would not be posting here.

When the black magic of automatic initramfs lets you down, you ned to fall back on the manual controls.

About the only way te debug what is happening, is to sprinkle the init script in the initfamfs with commands to show mhats happening and pauses to let you read the console.

Please post the init script from your initrd and I'll suggest some debug additions.

If you made your own directory structure, that would be useful too.

----------

## triquetra

Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.

```
#!/bin/busybox sh

 

rescue_shell() {

    echo $1

    busybox --install -s

    exec /bin/sh

}

# Mount the /proc and /sys filesystems.

mount -t devtmpfs none /dev  

mount -t proc none /proc

mount -t sysfs none /sys

# Assemble RAID:

( sleep 2

  mdadm --assemble --scan

  sleep 2

) &

wait

# Mount the root filesystem.

mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell"

    

# Clean up.

umount /proc

umount /sys

umount /dev

     

# Boot the real thing.

exec switch_root /mnt/root /sbin/init

```

I have made my own directory structure as follows:

```

.

./bin

./bin/busybox

./lib64

./dev

./dev/null

./dev/tty

./dev/console

./dev/md126p5

./root

./proc

./mnt

./mnt/root

./sbin

./sbin/mdadm

./lib

./init

./etc

./etc/mdadm.conf

./sys

7 blocks

```

Thank you in advance for your help.

----------

## NeddySeagoon

triquetra,

That looks "mostly harmless" and in line with the accepted wisdom.

I had similar pain a few months ago trying to get a remote box up.  In the end I spent a few days debugging locally.

What I found was the busybox no longer runs its own applets without a full path and or symlinks. 

This means your 

```
# Mount the /proc and /sys filesystems.

mount -t devtmpfs none /dev 

mount -t proc none /proc

mount -t sysfs none /sys 
```

all fail, then its steadily downhill from there.

Its not until you get to 

```
|| rescue_shell "Problem mounting root.  Dropping to shell" 
```

that the symlinks have a chance to get written.  That may fail too as  busybox is at /bin/busybox and no PATH is set.

Heres a few snippits of my init.  Notice the full paths.

```
#!/bin/busybox sh

rescue_shell() {

    echo "$@"

    echo "Something went wrong. Dropping you to a shell."

# have time to read the message

    /bin/sleep 20 

    /bin/busybox --install -s

    exec /bin/sh

}
```

and the commented out debug

```
#echo "/bin"

#/bin/ls -l /bin

#echo "/sbin"

#/bin/ls -l /sbin

#/bin/sleep 20

#echo "Mount proc"

/bin/mount -t proc proc /proc

#/bin/busybox sleep 20
```

/bin/ls is a symlink to /bin/busybox that is created elsewhere, likewise /bin/sleep.

You will need symlinks for all of the busybox applets you want to use.

You can either set the PATH or use full path names.  Setting the PATH was something I got wrong once, so I just left it in a mess.

----------

## triquetra

Hmmm ... no dice.

Here's my current init script:

```
#!/bin/busybox sh

 

# Create applets

/bin/busybox --install -s

pause() {

    read -p "$*"

}

rescue_shell() {

    /bin/echo $1

    /bin/busybox --install -s

    exec /bin/sh

}

# Mount the /proc and /sys filesystems.

/bin/mount -t devtmpfs none /dev  

/bin/mount -t proc none /proc

/bin/mount -t sysfs none /sys

#echo /sbin/mdev > /proc/sys/kernel/hotplug

#mdev -s

# Assemble RAID:

( sleep 2

  /sbin/mdadm --assemble --scan

  sleep 2

) &

wait

pause "raid array just assembled.  Press Enter to continue"

# Mount the root filesystem.

/bin/mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell"

    

# Clean up.

/bin/umount /proc

/bin/umount /sys

/bin/umount /dev

     

# Boot the real thing.

exec /sbin/switch_root /mnt/root /sbin/init

```

And here's what it gives me.

----------

## NeddySeagoon

triquetra,

I don't see whats happening with raid assembly but /dev/sdb unknown partition table looks odd.

I can't see what it says about /dev/sda

This suggests that you are using either fakeraid in raid0 across sda and sdb, in which case you only get a partition table on sda, or that you are using a partition table that is not supported by your kernel.

The kernel can see the HDD now, so thats progress.

----------

## triquetra

Yeah, some of the text is missing, and it scrolls by too fast for me to get it.  Because of the stage that bootup is failing, I presume that there is no text file for me to obtain the missing text, right?

I am using mdadm with a fakeraid raid0 setup across sda and sdb.  When the computer boots (with my old initramfs created with the old genkernel), it mounts at /dev/imsm0 (or something similar).  Why would I get a partition table on sdb with genkernel initramfs, but not with the custom initramfs?

I'll try to post the genkernel-next boot failure screen for comparison a bit later.

----------

## triquetra

The change in my custom init script did not change the result.  The kernel panic screen looks nearly identical, if not exactly so, in comparison to my previous custom initramfs.

Here's a pic of the genkernel-next failure for comparison.  That seems to get farther along than the custom initramfs, but still no boot.

----------

## triquetra

Still no progress here.  Any more ideas?

----------

## s0be

Is there any chance you're missing the devicemapper or raid modules in your kernel?  If you compiled them as modules and put them in your init.d, you would probably need to manually insmod them.

----------

## triquetra

Devicemapper and raid modules are built into the kernel.

----------

## juggling_ben

Few things:

Can you post a ls -la from inside your init directory structure? Your image shows an error code of -40 when running init, which according to my reading of the kernel code, means it can't determine the binary format of /init

Also, in your listing of your initramfs, you seem to be missing any .so files. Are you compiling the tools with the 'static' use flag? Particularly busybox.

I'd be happy to share by init script and my little build script if you'd like. It's only slightly more complicated than yours (after assembling the raid, I have to decrypt, do lvm stuff, etc) but hasn't failed me in quite some time.

----------

## s0be

Comparing your initrd to mine (ignoring the structural differences), you don't specify a mount point when you mount the gentoo named partition:

```

/bin/mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell" 

```

vs

```

mount -o ro $(findfs "$dev") /mnt/root >>/tmp/initrdlog 2>&1

```

in mine

You can see my initrd stuff at:

https://github.com/s0be/modular_initrd/blob/master/init

----------

## triquetra

 *juggling_ben wrote:*   

> Few things:
> 
> Can you post a ls -la from inside your init directory structure? Your image shows an error code of -40 when running init, which according to my reading of the kernel code, means it can't determine the binary format of /init
> 
> Also, in your listing of your initramfs, you seem to be missing any .so files. Are you compiling the tools with the 'static' use flag? Particularly busybox.
> ...

 

```
total 56K

drwxr-xr-x 13 root root 4.0K Feb 10 16:01 .

drwxr-xr-x 11 root root 4.0K Feb 20 10:14 ..

drwxr-xr-x  2 root root 4.0K Feb  8 16:32 bin

drwxr-xr-x  2 root root 4.0K Feb  9 14:16 dev

drwxr-xr-x  2 root root 4.0K Feb  9 13:56 etc

drwxr-xr-x  2 root root 4.0K Feb  8 16:27 lib

drwxr-xr-x  2 root root 4.0K Feb  8 16:27 lib64

drwxr-xr-x  3 root root 4.0K Feb  8 16:27 mnt

drwxr-xr-x  2 root root 4.0K Feb  8 16:27 proc

drwxr-xr-x  2 root root 4.0K Feb  8 16:27 root

drwxr-xr-x  2 root root 4.0K Feb  9 14:15 sbin

drwxr-xr-x  2 root root 4.0K Feb  8 16:27 sys

drwxr-xr-x  4 root root 4.0K Feb 10 16:01 usr

-rwxr-xr-x  1 root root  786 Feb 22 18:03 init
```

```
 * Found these USE flags for sys-apps/busybox-1.23.1-r1:

 U I

 - - debug         : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml

 + + ipv6          : Add support for IP version 6

 - - livecd        : !!internal use only!! DO NOT SET THIS FLAG YOURSELF!, used during livecd building

 - - make-symlinks : Create all the appropriate symlinks in /bin and /sbin.

 - - math          : Enable math support in gawk (requires libm)

 - - mdev          : Create the appropriate symlink in /sbin and install mdev.conf and support files

 + + pam           : Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip

 - - savedconfig   : Use this to restore your config from /etc/portage/savedconfig ${CATEGORY}/${PN}. Make sure your USE flags allow for appropriate dependencies

 - - sep-usr       : Support a separate /usr without needing an initramfs by booting with init=/ginit

 + + static        : Make the system rescue shell (/bin/bb) static so you can recover even when glibc is broken

 - - syslog        : Enable support for syslog

 + + systemd       : Support systemd

```

```
 * Found these USE flags for sys-fs/mdadm-3.3.2:

 U I

 + + static : !!do not set this during bootstrap!! Causes binaries to be statically linked instead of dynamically

```

Please share your init and build script.

----------

## triquetra

@s0be

Thanks for the reply, and sharing your initramfs.  I do usually find that problems I have with gentoo are because I've overlooked something simple, but specifying the mount point on the mount line was not, unfortunately, the problem (or at least not the primary one).  I changed my init to specify the mount point of /mnt/root, recompiled the initramfs, and rebooted, but got the same kernel panic.

----------

## s0be

your error code (-40) looks like it could be:

#define ELOOP           40      /* Too many symbolic links encountered */

So I bet you have some broken symbolic links once you consider that the top level of your init tree becomes / in initramfs.

The output of:

```

for f in $(find -type l); do ls -l $f; done;

```

Will show what your symlinks are.  They should all be 'relative' so that they work from both initramfs and when you're editing it.

```

s0be@anvil /usr/src/initramfs $ for f in $(find -type l); do ls -l $f; done;

lrwxrwxrwx 1 s0be s0be 3 Oct 11  2012 ./sbin -> bin

lrwxrwxrwx 1 s0be s0be 8 Oct 11  2012 ./usr/sbin -> ../sbin/

lrwxrwxrwx 1 s0be s0be 6 Oct 11  2012 ./usr/bin -> ../bin

```

----------

## juggling_ben

So thinking about this a bit, the problem isn't related to your init script. Your boot process isn't getting that far.

 *triquetra wrote:*   

> Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.
> 
> ```
> 
> .
> ...

 

7 blocks? That does not sound right to me. If I only include statically-linked busybox in my own, it is > 5000 blocks. What is bin/busybox in your initramfs? (ie what does "file bin/busybox" say?) If you copied a symlink or something instead, it would fit with my earlier confusion about the 'error -40'

----------

## s0be

 *juggling_ben wrote:*   

> So thinking about this a bit, the problem isn't related to your init script. Your boot process isn't getting that far.
> 
>  *triquetra wrote:*   Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.
> 
> ```
> ...

 

That makes a lot of sense, I bet $INITRDPATH/bin/busybox is a symlink to /bin/busybox.  That'd lead the the error -40 (ELOOP) as it'd end up being a symlink to itself, and it would give him the tiny initrd rather than one of appreciable size.

----------

## triquetra

Yes, that was it.  Thanks for the help debugging this issue.  I thought I would be clever and symlink the files needed for the initramfs so that I was sure to always have the current version.  Obviously, that did not work.  Would hard links accomplish that goal, or is it better to just use a mkinitramfs script to build the initramfs files and directory structure each time I rebuild the initramfs?

----------

## NeddySeagoon

triquetra,

Provided that there are no kernel modules in the initrd, its just like fimware.

Why would you ever rebuild it?

Its a closed system that does some stuff and gets abandoned once it done.

----------

## triquetra

@NeddySeagoon

Good question.  I had just gotten into the habit (when using genkernel) of building a new initramfs every time I upgraded my kernel.

----------

## s0be

 *triquetra wrote:*   

> Yes, that was it.  Thanks for the help debugging this issue.  I thought I would be clever and symlink the files needed for the initramfs so that I was sure to always have the current version.  Obviously, that did not work.  Would hard links accomplish that goal, or is it better to just use a mkinitramfs script to build the initramfs files and directory structure each time I rebuild the initramfs?

 

When it comes to initramfs, etc, making changes as rarely as possible is better, IMHO.  You don't always want the 'latest and greatest' versions in it, because they change behaviors and whatnot, and pre-init is a very delicate time.  I haven't upgraded the busybox in my initramfs since May 14, 2012.  If it works, don't replace it  :Wink: .

----------

## Hu

Other posters make a good argument for why not to rebuild your initramfs.  If you decide you want to rebuild it regularly, I recommend letting the kernel build system create it and embed it into the kernel.  This has the disadvantage of keeping one copy of the initramfs for each kernel you build, but the advantage that each kernel is wholly self-contained and has an initramfs loaded even if you use a bootloader that does not support loading an initramfs.

----------

## s0be

I didn't want to suggest embedding it directly..... but....

It's exactly what I do.  There's a problem with rebuilding kernels with embedded initramfs, here's the patch I've been using for years:

```

diff --git a/usr/Makefile b/usr/Makefile

index e767f01..041333b 100644

--- a/usr/Makefile

+++ b/usr/Makefile

@@ -60,15 +60,7 @@ targets := initramfs_data.cpio.gz initramfs_data.cpio.bz2 \

        initramfs_data.cpio.lzma initramfs_data.cpio.xz \

        initramfs_data.cpio.lzo initramfs_data.cpio.lz4 \

        initramfs_data.cpio

-# do not try to update files included in initramfs

-$(deps_initramfs): ;

-

-$(deps_initramfs): klibcdirs

-# We rebuild initramfs_data.cpio if:

-# 1) Any included file is newer then initramfs_data.cpio

-# 2) There are changes in which files are included (added or deleted)

-# 3) If gen_init_cpio are newer than initramfs_data.cpio

-# 4) arguments to gen_initramfs.sh changes

-$(obj)/initramfs_data.cpio$(suffix_y): $(obj)/gen_init_cpio $(deps_initramfs) klibcdirs

+

+$(obj)/initramfs_data.cpio$(suffix_y): $(obj)/gen_init_cpio

        $(Q)$(initramfs) -l $(ramfs-input) > $(obj)/.initramfs_data.cpio.d

        $(call if_changed,initfs)

```

to work around the issue.

Here's the error you'll see:

```

usr/Makefile:64: *** target pattern contains no '%'.  Stop.

Makefile:964: recipe for target 'usr' failed

make: *** [usr] Error 2

```

This patch might already be included in gentoo-sources, I built linux-next kernels semi-regularly as I do a lot of dev against it.

----------

