# zstd kernel compression error 11

## DeIM

Hi,

just tried to update kernel to 5.10.27 on old eeepc x86 laptop with 2GB RAM

I get:

```
  ZSTD22  arch/x86/boot/compressed/vmlinux.bin.zst

zstd: error 11 : Allocation error : not enough memory

make[2]: *** [arch/x86/boot/compressed/Makefile:134: arch/x86/boot/compressed/vmlinux.bin.zst]
```

The same on AMD 3000+ x64 with 2GB also with no problems at all.

Also tried to update zstd to 1.5.0 with no change.

Thanks

----------

## Perfect Gentleman

try another compression method

----------

## DeIM

tried lzo and it's OK

----------

## Perfect Gentleman

ZSTD consumes more RAM than your PC has

----------

## DeIM

strange - I've just built kernel on another 2GB RAM PC and ZSTD22 took 35% RAM at max.

Would zstd flag on gcc make the trick or it doesn't depend?

----------

## DeIM

the atom is HT - so it's presented as 2 core. Maybe zstd uses two threads and therefore more RAM. I'll try to disable threads flag on zstd.

----------

## Perfect Gentleman

I don't know for sure how much RAM it consumes in your case, but I can tell that on my PC with -j9 and 16GB it consumes about 5GB.

----------

## CaptainBlood

```
find . -name vmlinux -exec ls -la  {} \; | sort -k 9,9

-rwxr-xr-x 1 root root 3300152 juin  14 02:45 ./linux-4.14.235-gentoo-rt/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 32856720 juin  14 02:45 ./linux-4.14.235-gentoo-rt/vmlinux

-rwxr-xr-x 1 root root 3484472 juin  14 09:41 ./linux-4.14.236-gentoo-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 32253160 juin  14 09:41 ./linux-4.14.236-gentoo-classic/vmlinux

-rwxr-xr-x 1 root root 3624504 juin  22 20:19 ./linux-4.19.195-gentoo-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 32919256 juin  22 20:19 ./linux-4.19.195-gentoo-classic/vmlinux

-rwxr-xr-x 1 root root 3628600 juin  22 23:48 ./linux-4.19.195-gentoo-iptables/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 32921400 juin  22 23:48 ./linux-4.19.195-gentoo-iptables/vmlinux

-rwxr-xr-x 1 root root 3640888 juin  22 19:50 ./linux-4.19.195-gentoo-rt/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 33264848 juin  22 19:50 ./linux-4.19.195-gentoo-rt/vmlinux

-rwxr-xr-x 1 root root 3827360 juin  16 19:37 ./linux-5.10.41-gentoo-rt/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38179080 juin  16 19:37 ./linux-5.10.41-gentoo-rt/vmlinux

-rwxr-xr-x 1 root root 3807448 juin  23 22:26 ./linux-5.10.46-gentoo-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 37834328 juin  23 22:26 ./linux-5.10.46-gentoo-classic/vmlinux

-rwxr-xr-x 1 root root 3807032 juin  23 22:46 ./linux-5.10.46-gentoo-iptables/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 37830464 juin  23 22:45 ./linux-5.10.46-gentoo-iptables/vmlinux

-rwxr-xr-x 1 root root 3983304 mai   24 20:37 ./linux-5.11.22-gentoo-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38065824 mai   24 20:36 ./linux-5.11.22-gentoo-classic/vmlinux

-rwxr-xr-x 1 root root 3984720 mai   24 20:12 ./linux-5.11.22-gentoo-iptables/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38079472 mai   24 20:11 ./linux-5.11.22-gentoo-iptables/vmlinux

-rwxr-xr-x 1 root root 4039120 mars  12 11:42 ./linux-5.11.4-gentoo-rt/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38542888 mars  12 11:42 ./linux-5.11.4-gentoo-rt/vmlinux

-rwxr-xr-x 1 root root 4001416 juin  23 21:33 ./linux-5.12.13-gentoo-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38198056 juin  23 21:33 ./linux-5.12.13-gentoo-classic/vmlinux

-rwxr-xr-x 1 root root 4002280 juin  23 22:06 ./linux-5.12.13-gentoo-iptables/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38203336 juin  23 22:06 ./linux-5.12.13-gentoo-iptables/vmlinux

-rwxr-xr-x 1 root root 4066296 juin  30 01:12 ./linux-5.13.0-gentoo-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38906176 juin  30 01:12 ./linux-5.13.0-gentoo-classic/vmlinux

-rwxr-xr-x 1 root root 4100856 juin  29 03:43 ./linux-5.13.0-gentoo-iptables/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 38904728 juin  29 03:43 ./linux-5.13.0-gentoo-iptables/vmlinux

-rwxr-xr-x 1 root root 3661360 juin  14 01:40 ./linux-5.4.123-gentoo-rt/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 34125888 juin  14 01:40 ./linux-5.4.123-gentoo-rt/vmlinux

-rwxr-xr-x 1 root root 3636624 juin  14 01:51 ./linux-5.4.125-gentoo-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 33648688 juin  14 01:51 ./linux-5.4.125-gentoo-classic/vmlinux

-rwxr-xr-x 1 root root 3649072 juin  14 02:03 ./linux-5.4.125-gentoo-iptables/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 33734848 juin  14 02:03 ./linux-5.4.125-gentoo-iptables/vmlinux

-rwxr-xr-x 1 root root 3798960 juin  14 10:04 ./linux-5.8.0-lto-experimental-classic/arch/x86/boot/compressed/vmlinux

-rwxr-xr-x 1 root root 28747232 juin  14 10:04 ./linux-5.8.0-lto-experimental-classic/vmlinux
```

Maximized module configuration here.

Not using zstd with kernel here, favoring storage size vs speed using xz.

Quite suprised ~2Gb RAM get satured compressing ~35Mb file.

Could be so still, I don"t know zstd internal.  :Wink: 

Thks 4 ur attention, interest & support.

----------

## Hu

Some compression algorithms are very sensitive to the data being compressed, and may have very different memory requirements depending on the input data.  If your successful system was using a different kernel configuration, it may have presented a sufficiently different input that the memory requirements were lower.

----------

## DeIM

I thought kernel compression is just compression of uncompressed "blob". Seems zstd compression of kernel is preset to level 22.

I tried to manually compress vmlinux.bin with zstd level 22 on the same system where kernel built with zstd compression failed. It took ~20 % or RAM and successfully proceeded   :Shocked:  .

----------

## Perfect Gentleman

How many threads were used during manual compression?

----------

## Goverp

 *DeIM wrote:*   

> I thought kernel compression is just compression of uncompressed "blob". Seems zstd compression of kernel is preset to level 22.
> 
> I tried to manually compress vmlinux.bin with zstd level 22 on the same system where kernel built with zstd compression failed. It took ~20 % or RAM and successfully proceeded   .

 

I found https://github.com/facebook/zstd/issues/435, which sort of implies level 22 is for 64-bit machines, and IIUC this is a 32 bit box.

----------

## DeIM

Thanks for reply, I have progress:

Kernel compression does sth. like this:

```
# cat vmlinux.bin vmlinux.relocs | zstd -f --ultra -22 -T1 - -o test.zst

zstd: error 11 : Allocation error : not enough memory

# cat vmlinux.bin vmlinux.relocs > test.bin

# zstd -f --ultra -22 -T1 test.bin -o test.zst

test.bin             : 39.16%   (17709000 => 6934623 bytes, test.zst)
```

last command took ~400 MiB of RAM.

It seem to me the kernel's way rely on zstd's checks which isn't correct in this situation.

----------

## DeIM

seem this could be the way to set specific zstd level:

https://patchwork.kernel.org/project/linux-kbuild/patch/1637052983.69379.1617975065141@office.mailbox.org/

----------

## DeIM

OK, I just hacked current kernel Makefile:

```
$(call if_changed,zstd22)

->

$(call if_changed,zstd)
```

which used this command:

```
cat arch/x86/boot/compressed/vmlinux.bin arch/x86/boot/compressed/vmlinux.relocs | zstd -19
```

compression consumed ~100 MiB of RAM.

Difference in size:

```
6934623 zstd22

6940135 zstd (19)

9010421 lzo
```

I just wonder if kernel default compression level 22 could do any harm on 32 bit PC.

----------

