# Gentoo freezes when it runs out of RAM

## mikexx

Hi,

my laptop is very slow/freezes when it runs out of memory. The laptop has 16GB RAM and 16GB swap (SSD). However, as soon as ca 15 GB (RAM) are used the system freezes, for example moving the mouse pointer or enter the password in the lock screen takes more than one hour. I can observe that when chromium is emerged or when several VMs are open. 

I am posting that in the kernel forum as I think it is because of a kernel configuration. 

Any ideas?

Best

mike

----------

## 1clue

What do the logs say? Is there anything in any logs concurrent with the frozen state? Or before?

As well, have you definitely linked this to an out-of-memory scenario in more than one instance?

I have a Gentoo box with the same memory, with some amount of swap configured (don't know off hand) but it does not have this problem.

I would suspect that, if it's actually swap-related, there would be some sort of log messages about swap. Or disk access. Or, if it's related to something else you should get messages about that. The fact that you still get extremely slow activity means it's not a kernel panic, so there should be a log message somewhere.

What comes to mind is an extreme thrashing situation. Or deadlocks without a means to break out.

----------

## mikexx

I didn't see  anything suspicious last time. I will check it again. 

Best

mike

----------

## mikexx

I've checked the kernel logs again. Nothing has been logged about swap or something else what could explain the behavior.

Best

mike

----------

## guitou

Hello.

Most probably a silly question, but did you double check that your swap part is active?

++

Gi)

----------

## P.Kosunen

All systems freeze when swapping start. You can disable swap to not to freeze, but then some processes die when you run out of memory.

----------

## mikexx

@guitou

Swap is active. 

@P.Kosunen

I am almost sure that I used systems in the past that did not freeze that way. 

Best

mike

----------

## 1clue

Re: freezing when swapping starts, it's usually transparent to the user. Operations that normally happen in RAM are stopped, and disk activity happens to page RAM out to the swap device. 

If your system has a noticeable lag when swap starts there's a problem, or you have a really slow system in the first place, and a slow disk. If the system gets into a really bad place, swapping out huge amounts of RAM to disk and swapping other huge amounts back, then you can get noticeable lag. If the system is about to fail in spite of the swap, the lag can be severe.

That said, the very premise of swap is simply a last-ditch effort to keep a process working even though memory ran out. It's supposed to be a rare thing, not something that happens all the time. If you frequently have swap activity on your system you will get significant performance hits on long-running processes, and you will also have a higher probability of disk failure.

----------

## PCmaniaK

My two cents: Do not USE="jumbo-build" emerging chromium. It takes all the memory and asks for more.

----------

## mikexx

 *1clue wrote:*   

> Re: freezing when swapping starts, it's usually transparent to the user. Operations that normally happen in RAM are stopped, and disk activity happens to page RAM out to the swap device. 
> 
> If your system has a noticeable lag when swap starts there's a problem, or you have a really slow system in the first place, and a slow disk. If the system gets into a really bad place, swapping out huge amounts of RAM to disk and swapping other huge amounts back, then you can get noticeable lag. If the system is about to fail in spite of the swap, the lag can be severe.
> 
> That said, the very premise of swap is simply a last-ditch effort to keep a process working even though memory ran out. It's supposed to be a rare thing, not something that happens all the time. If you frequently have swap activity on your system you will get significant performance hits on long-running processes, and you will also have a higher probability of disk failure.

 

My system should not be very slow. It is a Thinkpad T480 with an i7, 16GB RAM and 512 GB SSD. Sure, it is normal that systems get slow as soon as swap starts, however, it should not take more than one hour to unlock the lock screen. It does not happen very often, only during system updates or if I have to start several virtual machines and because of other processes the system runs out of memory. 

cheers

mike

----------

## C5ace

Run sys-process/iotop in a terminal and see which process reads and writes from and to disk.

----------

## Ant P.

What disk scheduler are you using? (grep '' /sys/block/*/queue/scheduler)

----------

## Goverp

I'd say you need to dissect /proc/meminfo.  I've been reading up on things like HugePage support, which sounds great, but HugePages are unswappable.  (Not saying this has anything to do with this particular situation, just that there be sharks in them there waters.  I'm a complete novice here.)  There are probably lots of other things that meminfo might be able to warn of.

One thing I found related to emerges - you'll find recommendations to use tmpfs for portage temporary files.  And it's great for small apps.  BUT if you use it for emerging big ones like Rust, webengine, LLVM, and chromium, it can be a disaster.  You end up with the system having to swap out the temporary files to make space for the compilation, which wants to read the temp files and so swaps them back in.  In essence, don't overcommit memory - tmpfs should be no more than say 50% of memory.  I expect this gets even worse if you use "-pipe" as also recommended.  Basically, for the big beasts, use package.env settings to switch OFF all the "go-faster" settings or your machine grinds to a halt.

Unfortunately this sort of problem becomes increasingly common - developers assume vast amounts of memory for cache, big data trees and so forth, and forget they are cohabiting with other apps doing the same.  Many years ago I used an IBM mainframe with less than 1MB memory running a multi-user VSE/CICS system under VM, and we had "balanced systems guidelines" that reflected best practice, and explained how to identify memory hogs.  These days I doubt even a keyboard poll routine would be rated "good".

----------

## mikexx

 *Ant P. wrote:*   

> What disk scheduler are you using? (grep '' /sys/block/*/queue/scheduler)

 

```

/sys/block/dm-0/queue/scheduler:none

/sys/block/dm-1/queue/scheduler:none

/sys/block/dm-2/queue/scheduler:none

/sys/block/dm-3/queue/scheduler:none

/sys/block/dm-4/queue/scheduler:none

/sys/block/loop0/queue/scheduler:[mq-deadline] kyber none

/sys/block/loop1/queue/scheduler:[mq-deadline] kyber none

/sys/block/loop2/queue/scheduler:[mq-deadline] kyber none

/sys/block/loop3/queue/scheduler:[mq-deadline] kyber none

/sys/block/loop4/queue/scheduler:[mq-deadline] kyber none

/sys/block/loop5/queue/scheduler:[mq-deadline] kyber none

/sys/block/loop6/queue/scheduler:[mq-deadline] kyber none

/sys/block/loop7/queue/scheduler:[mq-deadline] kyber none

/sys/block/nvme0n1/queue/scheduler:[none] mq-deadline kyber     # <- my ssd (encrypted)

/sys/block/sda/queue/scheduler:noop deadline [cfq] 

/sys/block/sdc/queue/scheduler:noop deadline [cfq] 
```

@Goverp

Thanks. I will look at that. 

Best

mike

----------

## tholin

When you say "my ssd (encrypted)" does that mean you have an encrypted swap partition with dm-crypt? If so that it likely the problem.

Swap on dm-crypt is problematic. I don't know for sure what the problem is but I suspect dm-crypt does memory allocations when data is encrypted. The problem with that is you are already low on memory when data is written to swap. If dm-crypt needs to do memory allocations before writing the data those allocations will force out even more data, which will result in even more memory allocations, which will force out even more data... [repeat]

The result is a long stall.

You can test if that is the case by reformatting the encrypted swap partition as a regular swap partition.

https://bugzilla.kernel.org/show_bug.cgi?id=200105

Here is a bug report I found while googling the problem a while back. There is no final conclusion in that report.

----------

## mikexx

 *tholin wrote:*   

> When you say "my ssd (encrypted)" does that mean you have an encrypted swap partition with dm-crypt? If so that it likely the problem.
> 
> Swap on dm-crypt is problematic. I don't know for sure what the problem is but I suspect dm-crypt does memory allocations when data is encrypted. The problem with that is you are already low on memory when data is written to swap. If dm-crypt needs to do memory allocations before writing the data those allocations will force out even more data, which will result in even more memory allocations, which will force out even more data... [repeat]
> 
> The result is a long stall.
> ...

 

When I wrote 'encrypted' I thought that could be the problem.  :Smile: 

Yes swap is encrypted with dm-crypt. 

Best

mike

----------

