# System freeze on low memory

## haarp

I know I'm not the first one to report this. Google finds a million articles about the subject, but none offer an actual solution. Maybe someone here knows something.

My machine has 24GB of RAM and twice that for swap. With this I can happily use memory-intensive applications and keep /tmp and other caches as tmpfs (which I assume just counts towards "used" in htop and friends). The system will barely touch the swap for some reason but works well.

Until I run out of RAM that is. Which can happen when running VMs or opening huge images in Gimp. When this happens, the system freezes. Badly.

Facts:

- System slows down to the point that not even the mouse pointer or panel clock will work. It will occasionally twitch and react to input, but only after minutes delay

- Very heavy HDD thrashing visible on the HDD led. Presumably endlessly paging back and forth code pages, which explains why code execution slows to a crawl

- Calling the OOM killer (SysRq-f) does nothing. It dutifully logs the invokement in the kernel log, but doesn't bother actually killing anything

- After a few minutes to hours, the system thaws. It carries on like nothing happened, other than now having a few GB in swap instead of physical RAM

This has happened about a dozen times now. OOM killer doesn't feel like getting invoked and swap only really gets used after a catastrophic freeze has occured. What can I do to make it actually use swap before everything goes to shit? vm.swappiness did not do the trick. I'd also like for the OOM killer to actually do something during a freeze condition.

I have the suspicion that my heavy use of tmpfs may contribute to the kernel not realizing it's running low, but this is only a theory. In any case, it should swap out tmpfs first...

I'm open for any ideas. Thanks!

----------

## NeddySeagoon

haarp,

Gimp does its own thing for swapping out image data. By default, its in /home/<username>/.config/GIMP/<ver>/

That's likely to be on your HDD somewhere.

Look under Edit/Preferences/Folders, which is at the bottom of the Preferences menu, just out of sight.

For GIMP to do its own thing, it would have to ask for some RAM and lock it, so the kernel would not try to swap it.

Over allocating RAM to VMs is a killer. The kernel is quite happy to swap out the dynamically allocated RAM in use by VMs.

I did manage to allocate 16G RAM to an assortment of VMs on a system with only 8G of physical RAM.

Its a lesson I won't forget. 

Swap is only used for dynamically allocated RAM. Anything that has a permanent home on HDD can be dropped and reloaded.

It may need to be written first. Dirty buffers are never written to swap.  

This dropping and reloading is a form of swapping

----------

## benchaney

I think you have understood the problem backwards. The huge freeze you are describing is most likely caused by swap. At some point your system runs out of physical memory and decides to move some pages to swap. The system is frozen because writing multiple gigabytes to disk takes time, and because nothing can be accomplished when the system cannot allocate memory. The solution you are looking for is a way to prevent your system from swapping rather than encouraging it to.

----------

## haarp

Neddy,

I know about Gimp's own swap space. The problem is more due to the fact that Gimp leaks memory like an oil well. Opening and closing images barely ever returns the memory that was used by them.

In any case, there is still enough used memory from other apps and tmpfs that can be swapped out. It's just not being done until a freeze, for some reason.

 *benchaney wrote:*   

> I think you have understood the problem backwards. The huge freeze you are describing is most likely caused by swap. At some point your system runs out of physical memory and decides to move some pages to swap. The system is frozen because writing multiple gigabytes to disk takes time, and because nothing can be accomplished when the system cannot allocate memory. The solution you are looking for is a way to prevent your system from swapping rather than encouraging it to.

 

I don't think this is the case. For one thing, swap is on a very fast SSD. Furthermore, the same effect can be witnessed when you run out of RAM without any swap at all. This isn't swap being written (a matter of seconds), it's page thrashing.

----------

## NeddySeagoon

haarp,

There is no fix for a memory leak other than fix the code.

A workaround is to restart the app.

----------

## haarp

Neddy,

Indeed. But I'm not in the market for fixed apps. I'm in the market for kernels that use swap and execute the OOM killer when necessary.

----------

## krinn

-> https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

 *Quote:*   

> 4) And probably a lot more I do not know about 
> 
> tmpfs has three mount options for sizing:
> 
> size:      The limit of allocated bytes for this tmpfs instance. The 
> ...

 

deadlock remind you someone?

----------

## haarp

This refers to hard oom condition.

From the same document:

 *Quote:*   

> tmpfs (...) is able to swap unneeded pages out to swap space.

 

So that can't be it. Or rather, the fact that it isn't swapping until it's too late is exactly it.

----------

## NeddySeagoon

haarp,

Have you tested that?

Put /tmp on the HDD and see if the problem goes away.

----------

## tholin

As you have noticed you are not alone with problems like these. 

 *haarp wrote:*   

> My machine has 24GB of RAM and twice that for swap.

 

So you have 48GB!?! swap? My best guess for what is happening on your system is that gimp suddenly makes a very large (a few GB) memory allocation and you run out of ram. The kernel tries to satisfy the allocation by writing out several GB of inactive anonymous memory to swap but the swap subsystem is crap so the writeout will be slow even on fast ssds. While the swapout is happening the system is still out of ram so programs trying to do memory allocation stalls waiting for that writeout.

The OOM killer is only activated when you are really out of ram+swap and filling 48GB of swap takes forever. Quite often the OOM killer isn't even activated even if you run out of ram+swap.

https://lwn.net/Articles/753840/

"First we used it on systems that frequently locked up in low memory situations. The reason this happens is that the OOM killer is triggered by reclaim not being able to make forward progress, but with fast flash devices there is *always* some clean and uptodate cache to reclaim; the OOM killer never kicks in, even as tasks wait 80-90% of the time faulting executables. There is no situation where this ever makes sense in practice."

What that means is that even if you reduce the size of your swap, even disabling it completely, you will probably still get stalls. I don't know of any good solution to this problem.

I don't know why manual invoked of the OOM killer doesn't work. Do you use Chrome? I've noticed that Chrome likes to self sacrifice by setting a high oom_score_adj value on some of it's processes. That means Chrome's process is more likely to be OOM killed even when its memory consumption is relatively low. The little memory that is freed by killing it is instantly used again to satisfy the large allocation by gimp and you are still out of memory. You can see in the syslog which process the OOM killer tried to kill.

----------

## P.Kosunen

Edit: Nevermind, did not read all.

----------

## bunder

would increasing min_free_kbytes help at all?  by default its pretty low actually, something like 11mb.  i'm experimenting with bumping it up to 64mb myself.

----------

## eccerr0r

I think the OOM killer is getting confused or not realizing the extent of the problem.

To the OOM killer, you still have memory available.  There's swap available.  You haven't completely exhausted all memory but have gotten to the point the system is unusable.

However there seems to be an issue with tmpfs: I've _never_ seen tmpfs get swapped out, as if it is locked in memory much like a VM.  I don't know if it's a bug or not but it's stuck in memory, even if it hasn't been recently used.

So the "fix" is probably to depend on VFS cache and not tmpfs (i.e. don't use tmpfs!!!) if you're working on large files and expect to overrun RAM when accounting for both tmpfs usage and regular memory usage.

----------

## haarp

 *bunder wrote:*   

> would increasing min_free_kbytes help at all?  by default its pretty low actually, something like 11mb.  i'm experimenting with bumping it up to 64mb myself.

 

From what I read about it, this only keeps some RAM free for interrupts and such. https://stackoverflow.com/a/26452693/5424487

 *eccerr0r wrote:*   

> I think the OOM killer is getting confused or not realizing the extent of the problem.
> 
> To the OOM killer, you still have memory available.  There's swap available.  You haven't completely exhausted all memory but have gotten to the point the system is unusable.
> 
> However there seems to be an issue with tmpfs: I've _never_ seen tmpfs get swapped out, as if it is locked in memory much like a VM.  I don't know if it's a bug or not but it's stuck in memory, even if it hasn't been recently used.
> ...

 

 *NeddySeagoon wrote:*   

> haarp,
> 
> Have you tested that?
> 
> Put /tmp on the HDD and see if the problem goes away.

 

I have not seen tmpfs ever use swap, either. I'm curious to know more.

Disabling it entirely could be a valid test,. I'll try that in the future.

 *tholin wrote:*   

> As you have noticed you are not alone with problems like these. 
> 
>  *haarp wrote:*   My machine has 24GB of RAM and twice that for swap. 
> 
> So you have 48GB!?! swap?

 

Not quite, it's 32GB. The reason being to allow suspend-to-disk, even if I upgrade to 32GB RAM in the future.

 *Quote:*   

> My best guess for what is happening on your system is that gimp suddenly makes a very large (a few GB) memory allocation and you run out of ram. The kernel tries to satisfy the allocation by writing out several GB of inactive anonymous memory to swap but the swap subsystem is crap so the writeout will be slow even on fast ssds. While the swapout is happening the system is still out of ram so programs trying to do memory allocation stalls waiting for that writeout.
> 
> The OOM killer is only activated when you are really out of ram+swap and filling 48GB of swap takes forever. Quite often the OOM killer isn't even activated even if you run out of ram+swap.

 

Mhh. I don't doubt that, but I've never ever even come close to filling swap. The highest it got was 2GB, and that was after one of these "OOM freezes".

 *Quote:*   

> https://lwn.net/Articles/753840/

 

Looks like an interesting concept. Unfortunately it seems CONFIG_PSI is not merged and in a very very experimental state.

 *Quote:*   

> I don't know why manual invoked of the OOM killer doesn't work. Do you use Chrome? I've noticed that Chrome likes to self sacrifice by setting a high oom_score_adj value on some of it's processes. That means Chrome's process is more likely to be OOM killed even when its memory consumption is relatively low. The little memory that is freed by killing it is instantly used again to satisfy the large allocation by gimp and you are still out of memory. You can see in the syslog which process the OOM killer tried to kill.

 

I can tell from the kernel log that it got invoked (three times, by me), but it didn't log killing any processes, nor were and programs missing after it thawed. I'd expect it to kill Gimp, or Firefox worker processes, but nope. OOM killer is a lazy bastard.

----------

## eccerr0r

Well the problem is that the biggest consumer of memory may not be gimp or firefox, it may be tmpfs -- which it can't "kill"!

I think there is sort of a "cultural" problem with people trying to outsmart cache by using tmpfs with the historical "ramdisk" methodology, trying to squeeze the last bit of speed.  Indeed there are situations where there is a win by using tmpfs -- that is, if the files there are really temporary.  But people copying work files into tmpfs to "work" on it is a flawed methodology on systems that share memory between the two.

I know that VMs should not swap out pages else it will become painfully obvious in a VM.  Also shared memory should not be swapped out either, there's some race conditions to deal with.  Tmpfs is commonly used for shared memory, so I can see why there's a policy not to swap it out...

----------

## haarp

 *eccerr0r wrote:*   

> Well the problem is that the biggest consumer of memory may not be gimp or firefox, it may be tmpfs -- which it can't "kill"!

 

This would be a valid kernel bug, no?  :Wink:  But it would surely complain if that was the case. The only logs I see from a manual OOM killer invocation are:

```
kernel: sysrq: SysRq : Manual OOM execution

kernel: Purging GPU memory, 57690 pages freed, 4158 pages still pinned.
```

 *Quote:*   

> I think there is sort of a "cultural" problem with people trying to outsmart cache by using tmpfs with the historical "ramdisk" methodology, trying to squeeze the last bit of speed.  Indeed there are situations where there is a win by using tmpfs -- that is, if the files there are really temporary.  But people copying work files into tmpfs to "work" on it is a flawed methodology on systems that share memory between the two.

 

I quite like it for /tmp and ~/.cache. Not for speed, but for cleanliness. I use it as temporary workspace, staging area and a place where downloads land first. It forces me to organize and move important stuff to permanent storage. It's my way of keeping my fs clean and my head sane  :Smile: 

 *Quote:*   

> Also shared memory should not be swapped out either, there's some race conditions to deal with.  Tmpfs is commonly used for shared memory, so I can see why there's a policy not to swap it out...

 

Honestly, this sounds a bit farfetched. And do you mean actual shared memory (as seen in htop and friends) or /dev/shm, which is a hacky method of IPC?

Thank you everyone for your responses so far!

----------

## tholin

 *haarp wrote:*   

> The only logs I see from a manual OOM killer invocation are:
> 
> ```
> kernel: sysrq: SysRq : Manual OOM execution
> 
> ...

 

That reminds me of a bug report I saw a while back. Kernel subsystems can register a callback which will triggers on out of memory. Intel's gpu drivers use that for freeing up some "stolen" gpu memory when memory runs low. As long as any callback succeeded in freeing some memory the OOM killer is happy and considers the work done. Only problem is that the gpu drivers will instantly steal some memory back again for rendering. That's probably what happens when you manually invoke the OOM killer.

I can't find that bug report now and I don't know if it was ever fixed. What kernel are you using?

EDIT: Just tested on my 4.14.50 system and the OOM killer only frees gpu memory and nothing else. So that bug is not fixed and that explains why the OOM killer doesn't work.

```

[  +0.000005] Manual OOM execution

[  +0.006559] Purging GPU memory, 37528 pages freed, 6917 pages still pinned.

[ +17.281449] sysrq: SysRq : 

[  +0.000005] Manual OOM execution

[  +0.005957] Purging GPU memory, 29936 pages freed, 6917 pages still pinned.

[  +1.445043] sysrq: SysRq : 

[  +0.000004] Manual OOM execution

[  +0.003587] Purging GPU memory, 20427 pages freed, 6917 pages still pinned.

[  +0.976394] sysrq: SysRq : 

[  +0.000004] Manual OOM execution

[  +0.002739] Purging GPU memory, 15917 pages freed, 6917 pages still pinned.

[  +0.949256] sysrq: SysRq : 

[  +0.000005] Manual OOM execution

[  +0.003527] Purging GPU memory, 20411 pages freed, 6917 pages still pinned.

[  +0.768492] sysrq: SysRq : 

[  +0.000005] Manual OOM execution

[  +0.003171] Purging GPU memory, 20282 pages freed, 6917 pages still pinned.

[  +0.452819] sysrq: SysRq : 

[  +0.000005] Manual OOM execution

[  +0.001824] Purging GPU memory, 10344 pages freed, 6917 pages still pinned.

[  +0.338172] sysrq: SysRq : 

[  +0.000005] Manual OOM execution

[  +0.001669] Purging GPU memory, 11207 pages freed, 11417 pages still pinned.

[  +0.237308] sysrq: SysRq : 

[  +0.000004] Manual OOM execution

[  +0.001364] Purging GPU memory, 9079 pages freed, 6917 pages still pinned.

```

----------

## eccerr0r

I've found that gentoo-sources-4.14.11-r2 actually seems to work as expected, at least if I specify a tmpfs that exceeds physical memory, it does allocate space in swap, so at least this portion is working as expected...  

However, gentoo-sources-4.9.95 and 4.12.12 -- it wedges my other machine if I try to allocate a file in tmpfs that exceeds available physical memory.

Different machines however, so may not be comparing the same thing.  

If anyone wants to try: first make sure you don't have anything useful running in case your machine dies.

Make sure you have plenty of swap for this experiment.  Linux has no options but to page text if it has no swap which kills.

If you have a consolekit or systemd-logind machine, /tmp and /run/user/${UID} are both writeable by the user.  If "lucky" both are half the machine's RAM, you just need to create files that sum up to your physical memory.  Create a file that fills both of them:

$ cat /dev/zero > /tmp/garbage1

$ cat /dev/zero > /run/user/${UID}/garbage1

If your machine starts getting wedged (mouse stops moving, can't type, etc., etc.) when the second one is running, you have a defective kernel...  The expected result should be programs reporting disk full not system failure!

I don't know if control groups or memory limits will prevent a user from letting the machine crash when filling up RAM/tmpfs.

----------

## haarp

Just a small update on this.

@tholin, yes, this is likely what made the OOM killer ineffective. In any case, it wasn't automatically triggered to begin with, which is the main issue. Or was.

I upgraded from 24GB to 32GB RAM. Whoever said that throwing hardware at problems never solved anything? For now I'm saved from it.

I discovered oomd, which Facebook just open-sourced. Might be just what I need should the issue ever return.

https://lwn.net/Articles/760461/

----------

