# invalid opcode in busybox [SOLVED]

## xdarma

After a cpu change, new kernels fails on boot with this error:

```
traps: init[1] trap invalid opcode ip:4aef39 sp:7ffe99ad4c90 error:0 in busybox[400000+232000]
```

This happens with 4.17.x, 4.18,x and 4.19.x also using different config.

So I think it's a problem NOT connected to kernel.

All packages was rebuilt before cpu change with "-march=x86-64".

This is my should-be-safe busybox configuration:

```

aquilante ~ # genlop -i sys-apps/busybox

 * sys-apps/busybox

   Total builds: 37

   Global build time: 28 minutes and 37 seconds.

   Average merge time: 46 seconds.

   Info about currently installed ebuild:

   * sys-apps/busybox-1.29.0

   Install date: Mon Nov 26 11:03:22 2018

   USE="ipv6 -static -debug -livecd -make-symlinks -math -mdev -pam -selinux -sep-usr -syslog -systemd"

   CFLAGS="-O2 -march=x86-64 -pipe -fno-strict-aliasing   CXXFLAGS="-O2 -march=x86-64 -pipe -fvisibility-inlines-hidden -fno-strict-aliasing   LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"

```

Very likely, there is something I miss, but what?

TIA for any suggestion.

----------

## NeddySeagoon

xdarma,

"-march=x86-64" is not CPU safe.  

Don't set -march at all. Use "-mtune=generic" to get run anywhere AMD/Intel 64 bit code.

----------

## tholin

Do you use an initramfs and does it contain a static copy of busybox? If so make sure that static copy is also recompiled.

----------

## xdarma

Hi NeddySeagoon,

I was aware that -tune=generic has wider compatibility than -march=x86-64 but not that busybox is so "fragile".

New flags for special packages are:

-O2 -tune=generic

Other than busybox, there are other packages involved that is better to recompile?

Hi tholin,

yes, I'm using initramfs, generated by genkernel. So I guess there is a static copy in, but not sure about.

Thanks again for help.

----------

## Tony0945

 *NeddySeagoon wrote:*   

> Don't set -march at all. Use "-mtune=generic" to get run anywhere AMD/Intel 64 bit code.

 

When I change CPU's I usually run emerge -e world with march=native after the change.

Do I understand correctly that I should be running mtune=generic, no march and emerge -e BEFORE switching the CPU and my usual procedure afterward?

----------

## tholin

@Tony0945 Your new cpu might not support all cpu instructions the old one supported. If that's the case you'll get invalid opcode when you try to boot. If you are lucky the new cpu supports all old instructions and in that case everything works fine.

To avoid problems it better to rebuild everything with generic instructions before switching the cpu.

@xdarma Try to regenerate the initramfs so it contains the busybox built with generic instructions. Forgetting about the binaries in the initramfs is an easy mistake to make. I've done that myself.

----------

## Tony0945

I think I've been lucky in that all my CPU's have been AMD and until the bulldozer series they were backward compatible. The future is not guaranteed and in one of the Ryzen threads, it was pointed out that Ryzen does not have some instructions that some bulldozer/excavator have.

It might be enough to change to generic flags and emerge -e @system taking the chance on possibly chrooting from sysrescuecd. On these big builds, I usually start them just before going to bed at night and they are (almost) done in the morning.

----------

## Anon-E-moose

Interesting blog on march vs mtune

https://lemire.me/blog/2018/07/25/it-is-more-complicated-than-i-thought-mtune-march-in-gcc/

----------

## xdarma

Thanks to all for informations.

This gentoo-box is a testing machine and was build with Phenom II and -march=native.

Then I've rebuilt @world with -march=x86-64 and installed a Ryzen.

Everything goes well. So I've rebuilt with -march=native.

New rebuild of @world with -march=x86-64 and change cpu to a Piledriver FX.

Now gentoo works and can keep himself update but do not build a bootable kernel.

Changed something on setup:

```
aquilante ~ # cat /etc/portage/env/safe-cflags 

CFLAGS="-O2 -mtune=generic -pipe"

CXXFLAGS="-O2 -mtune=generic -pipe -fvisibility-inlines-hidden"
```

```
aquilante ~ # grep safe /etc/portage/package.env 

app-arch/cpio                   safe-cflags

app-arch/gzip                   safe-cflags

sys-apps/busybox                safe-cflags

sys-apps/sysvinit               safe-cflags

sys-fs/udev-init-scripts        safe-cflags

sys-kernel/genkernel            safe-cflags
```

Rebuilt initramfs with Initramfs Guide

but no luck...

----------

## Tony0945

 *xdarma wrote:*   

> Now gentoo works and can keep himself update but do not build a bootable kernel.

 

The kernel doesn't use CFLAGS. You have to set one of the following in menuconfig:

```
# CONFIG_MK8 is not set

# CONFIG_MK8SSE3 is not set

# CONFIG_MK10 is not set

# CONFIG_MBARCELONA is not set

# CONFIG_MBOBCAT is not set

# CONFIG_MJAGUAR is not set

# CONFIG_MBULLDOZER is not set

# CONFIG_MPILEDRIVER is not set

# CONFIG_MSTEAMROLLER is not set

# CONFIG_MEXCAVATOR is not set

# CONFIG_MZEN is not set

# CONFIG_MPSC is not set

# CONFIG_MATOM is not set

# CONFIG_MCORE2 is not set

# CONFIG_MNEHALEM is not set

# CONFIG_MWESTMERE is not set

# CONFIG_MSILVERMONT is not set

# CONFIG_MSANDYBRIDGE is not set

# CONFIG_MIVYBRIDGE is not set

# CONFIG_MHASWELL is not set

# CONFIG_MBROADWELL is not set

# CONFIG_MSKYLAKE is not set

# CONFIG_MSKYLAKEX is not set

CONFIG_MNATIVE=y

```

 This under the "PROCESSOR_FAMILY" submenu under "PROCESSOR TYPE AND FEATURES" in menuconfig.

----------

## xdarma

The "experimental" USE flag in gentoo-sources is disabled, my kernel configuration is:

```

                               ( ) Opteron/Athlon64/Hammer/K8

                               ( ) Intel P4 / older Netburst based Xeon

                               ( ) Core 2/newer Xeon

                               ( ) Intel Atom

                               (X) Generic-x86-64

```

The good thing is that error still the same.

There is an elephant, somewhere, but I can't catch it. :-)

----------

## Tony0945

What happens when it fails to boot? Any error message?

----------

## xdarma

As said, it fails with:

```
traps: init[1] trap invalid opcode ip:4aef39 sp:7ffe99ad4c90 error:0 in busybox[400000+232000]
```

Searching online for the opcode gives me nothing.

Now I'm rebuilding @world with -mtune=generic -mfinger-crossed. ;-)

----------

## Tony0945

Oh, but that refers to an invalid opcode in busybox, not the kernel.

Do I understand you correctly that you get the message with some versions of the kernel and not with others?

Are both kernel versions built with the same config? Maybe one is at a higher debug level than another.

EDIT:

It's easier to just rebuild busy box, two ways:

1. quick and dirty

```
#Edit make .conf with the generic CFLAGS

. /etc/profile

emerge -a1v busybox

#edit make,conf to restore CFLAGS

```

2. permanent

Refer to https://wiki.gentoo.org/wiki//etc/portage/package.env

Create a file in /etc/portage/env that contains the desired CFLAGS  change. i.e. /etc/portage/busybox.conf

Add an entry to /etc/portage/package.env  

```
sys-apps/busy box busybox.conf
```

Now busybox will always be built with those CFLAGS and the rest of the system will be built with the CFLAGS from make.conf)Last edited by Tony0945 on Tue Nov 27, 2018 4:12 pm; edited 1 time in total

----------

## xdarma

 *Quote:*   

> 
> 
> This happens with 4.17.x, 4.18,x and 4.19.x also using different config.
> 
> So I think it's a problem NOT connected to kernel.
> ...

 

At the moment, I'm using the latest kernel compiled with former cpu: gentoo-sources-4.15.13.

----------

## Tony0945

Hi. I see you are online. Try the method called permanent in my edit above. Optimizing busybox that runs once doesn't have a terribly high priority anyway.

----------

## Hu

According to that message, and confirmed by your description that boot fails, the faulting process is busybox when it runs as init as part of the initramfs.  Therefore, it is not sufficient to recompile busybox.  You must also replace the busybox in your initramfs with the recompiled busybox.  I suspect you may have already fixed the busybox installed as part of your regular system.  However, since the fault is in the busybox built into the initramfs, fixing your installed busybox is insufficient.

----------

## xdarma

Still broken. Also after rebuilding @world with -mtune=generic

@Tony0945

Thanks for suggestion, I use already the permanent method for memory greedy packages during compiling.

@Hu

Correct. Boot fails with something like below:

```

Freeing unused kernel image memory: 1292K

Write protecting the kernel read-only data: 18432k

Freeing unused kernel image memory: 2008K

Freeing unused kernel image meory: 508K

Run /init as init process

traps: init[1] trap invalid opcode ip:4aef39 sp:7ffe99ad4c90 error:0 in busybox[400000+232000]

Kernel panic - not syncing: Attempted to kil init! exitcode=0x00000004

```

I compile a new kernel with this genkernel command line:

```
genkernel --mrproper --menuconfig --kernel-config=/etc/kernels/kernel-config-x86_64-4.19.4-gentoo.g1 --save-config --kernname=opencl --loglevel=1 --install all
```

So I think that initramfs should be renewed every time I launch the command.

Maybe better to try to build a initramfs from scratch. Hard job for my poor skills...

----------

## zooppoop

I had the same problem.  As described above Genkernel has it's own copy of Busybox which is different than the one that is installed on the system.  It will build it and store it in /var/cache/genkernel   

In general I don't know if a way to tell it to rebuild this if it was built with different CFLAGS so you need to manually remove this file and then build the initramfs again so that it will get build again.

Just posting here as I spent way to much time rebuilding and rebuilding world wondering why the system could not finish booting.

Important line to see from Genkernel is this make sure you see it compiling it.

```
* busybox: >> Using cache
```

[/code]

----------

## xdarma

I haven't solved my problem, but checking under /var/cache/genkernel I found:

- busybox-1.20.2-x86_64.tar.bz2 created on Feb 2016

- busybox-1.27.2-x86_64.tar.bz2 created on Agu 2018

Installed busybox is 1.29.3.

Next time, I will delete all cache to see if it works.

Thanks for hint.

----------

## NeddySeagoon

xdarma,

Its your initrd that needs to be rebuilt.

The kernel starts, mounts your initrd as the root filesystem and starts using busybox as the shell.

Busybox gives you an illegal instruction.

The kernel does not use make.conf to build but the initrd will.

Remake busybox for the initrd.

----------

## xdarma

Problem solved.

The culprits was:

- my laziness, to not build a custom initramfs "by hand";

- the cache of busybox that genkernel create under /var/cache/genkernel that went too advanced when I downgraded the hardware.

So I removed the cache of genkernel with:

```
rm -rf /var/cache/genkernel/*
```

Then built a new kernel via the usual genkernel string. He built a new busybox cache and insert in initramfs.

Many thanks to all people that helped me, specially zooppoop.

----------

