# System becomes unresponsive while compiling

## haarp

Greetings!

For a few months now I'm having problems with my system becoming slow and almost unresponsive while compiling big projects, such as Firefox.

This wasn't a problem in the past. Unfortunately I can't tell exactly when it started, nor which of the one million changes I've made since then could be responsible. I suspect it was a kernel update or kernel setting i inadvertendly changed - something CPU scheduling related? Or the result of CPU bug mitigations?

Factsheet:

- Ivy Bridge quadcore with hyperthreading (i7-3920XM)

- Kernel gentoo-4.19.75

- using CFS task scheduler (I think)

- SCHED_MC=y, CONFIG_SCHED_SMT=y

- Timer Frequency = 300Hz, CONFIG_PREEMPT_VOLUNTARY=y

- using deadline IO scheduler on an SSD

- /var/tmp/portage is entirely in tmpfs

- PORTAGE_NICENESS="15"

- PORTAGE_IONICE_COMMAND="ionice -c3 -p \${PID}"

- MAKEOPTS="-j8"

- plenty of spare RAM, no swapping taking place

- Load average during compilation reaches as high as 14

The problem is very evident in my running browser (Firefox), which always has some CPU background activity (I blame my dozens of open/loaded tabs) and spikes CPU usage when loading or interacting with sites. It becomes very slow. However it (nice=0) should always win over portage (nice=15).

It's frustating. Any ideas? Thanks!

----------

## mike155

Is this a follow-up to https://forums.gentoo.org/viewtopic-p-8243856.html or something completely new?

----------

## haarp

 *mike155 wrote:*   

> Is this a follow-up to https://forums.gentoo.org/viewtopic-p-8243856.html or something completely new?

 

I'm surprised someone remembers that thread  :Smile: 

It's completely new. I haven't had the problem described in the old thread since upgrading to 32GB. Simply throwing memory at your problems helps!

----------

## mike155

We had a couple of threads where users complained that their systems get unresponsive. That's why I looked back. If I rememeber correctly, we never found out what caused the issue...

The first thing you can do is: try to find out what actually triggers the problem. For example, is it the CPU load? What happens if you run 14 times 

```
while [ 1 ]; do let i=1; done &
```

This will give you a load of 14. Does your system get unresponsive after a few minutes? Monitor the CPU core temperature (coretemp) using lm-sensors. Does it rise above 65°C?

If it's not the CPU load: add disk I/O or memory access to the test. If you know what causes the problem and if you can reproduce it, it will be easy to find a solution.Last edited by mike155 on Sat Oct 12, 2019 10:37 pm; edited 1 time in total

----------

## Anon-E-moose

 *haarp wrote:*   

> 
> 
> - SCHED_MC=y, CONFIG_SCHED_SMT=y
> 
> - Timer Frequency = 300Hz, CONFIG_PREEMPT_VOLUNTARY=y
> ...

 

I turned off SCHED_MC and SCHED_SMT, as I found they didn't help on my system.

I also run the timer at 1000 and PREEMPT instead of PREEMPT_VOLUNTARY.

----------

## krinn

 *Anon-E-moose wrote:*   

> I also run the timer at 1000 and PREEMPT instead of PREEMPT_VOLUNTARY.

 

same, but with SCHED_MC=y, CONFIG_SCHED_SMT=y

----------

## ulenrich

 *Anon-E-moose wrote:*   

>  *haarp wrote:*   
> 
> - SCHED_MC=y, CONFIG_SCHED_SMT=y
> 
> - Timer Frequency = 300Hz, CONFIG_PREEMPT_VOLUNTARY=y
> ...

 

I used for a long time PREEMPT_VOLUNTARY and got hit of this unresponsiveness also. Yesterday I changed my .config and got a very responsive linux-5.3.6 now using: 

```
# zcat /proc/config.gz |grep PREEMPT

# CONFIG_PREEMPT_NONE is not set

# CONFIG_PREEMPT_VOLUNTARY is not set

CONFIG_PREEMPT=y

CONFIG_PREEMPT_COUNT=y

CONFIG_PREEMPTION=y

CONFIG_PREEMPT_RCU=y 
```

 So my calculation to get better compiling performanc with VOLUNTARY got me hit in the ass lately. Seems PREEMPT_VOLUNTARY is not an option for anybody any more.

----------

## haarp

I've been experimenting with the MuQSS process scheduler lately (ck's successor to BFS) and setting my compilation jobs to SCHED_BATCH. this seems to work well and I maintain interactivity during heavy compiles.

For some weird reason tho, having heavy compile jobs running now seems to block new processes from spawning. e.g. opening a new terminal window either doesn't spawn any window at all, or I won't get a shell, until I stop/pause the compile job that is. Already running shells are unaffected.

Weirder and weirder...Last edited by haarp on Tue Nov 12, 2019 1:57 pm; edited 1 time in total

----------

## mike155

I really doubt that the 'unresponsiveness' is caused by CONFIG_PREEMPT_VOLUNTARY. It might well be that changing CONFIG_PREEMPT_XXXX helps, but something else is wrong.

Please post your kernel config using wgetpaste.

----------

## haarp

I agree.

 *mike155 wrote:*   

> Please post your kernel config using wgetpaste.

 

here you go.

----------

## mike155

Thanks for your kernel config.

My machine has an i5-3570K CPU and I run Linux kernel 4.19.75. 

I compared your kernel config ('<') with mine ('>'). Below are the most important differences. There are more, but let's start with those.

```
< # Linux/x86 4.19.2-ck Kernel Configuration

---

> # Linux/x86 4.19.72 Kernel Configuration
```

4.19.2-ck? Please switch to the vanilla kernel, at least until the problem is solved.

```
< # CONFIG_NO_HZ is not set

< # CONFIG_PREEMPT_VOLUNTARY is not set

< CONFIG_PREEMPT=y

< CONFIG_PREEMPT_COUNT=y

---

> CONFIG_NO_HZ=y

> CONFIG_PREEMPT_VOLUNTARY=y

> # CONFIG_PREEMPT is not set
```

You may want to try CONFIG_NO_HZ=y

```
> CONFIG_SCHED_AUTOGROUP=y
```

Automatic process group scheduling is NOT enabled on your machine. You may want to try that. It's known to make a difference.

```
< CONFIG_ARCH_HIBERNATION_HEADER=y

< CONFIG_SUSPEND=y

< CONFIG_SUSPEND_FREEZER=y

< CONFIG_HIBERNATE_CALLBACKS=y

< CONFIG_HIBERNATION=y

< CONFIG_PM_STD_PARTITION=""

< CONFIG_PM_SLEEP=y

< CONFIG_PM_SLEEP_SMP=y

< # CONFIG_PM_AUTOSLEEP is not set

< CONFIG_PM_WAKELOCKS=y

< CONFIG_PM_WAKELOCKS_LIMIT=100

< CONFIG_PM_WAKELOCKS_GC=y

---

> # CONFIG_SUSPEND is not set

```

You could try to disable Hibernation and Suspend. I doubt that it will make a difference, but it's worth a try...

----------

## haarp

Thanks for your efforts!

 *mike155 wrote:*   

> 
> 
> 4.19.2-ck? Please switch to the vanilla kernel, at least until the problem is solved.

 

I was using vanilla kernels at the start of this thread. I only switched to -ck a few days ago to see if its improved process scheduler would help. See my reply earlier today. -ck uses the same kernel config with only minor differences due to enabling of MuQSS.

 *Quote:*   

> You may want to try CONFIG_NO_HZ=y

 

This is basically enabled through CONFIG_NO_HZ_IDLE=y:

```
CONFIG_NO_HZ

This is the old config entry that enables dynticks idle.

We keep it around for a little while to enforce backward

compatibility with older config files.
```

 *Quote:*   

> Automatic process group scheduling is NOT enabled on your machine. You may want to try that. It's known to make a difference.

 

It was enabled on the vanilla kernel which showed the problems described in the start post. It's disabled now because it's incompatible with MuQSS.

 *Quote:*   

> You could try to disable Hibernation and Suspend. I doubt that it will make a difference, but it's worth a try...

 

I don't see how these could hurt. They're also features I need every day, so disabling them is not an option.

----------

## haarp

The problem I had in

 *haarp wrote:*   

> I've been experimenting with the MuQSS process scheduler lately (ck's successor to BFS) and setting my compilation jobs to SCHED_BATCH. this seems to work well and I maintain interactivity during heavy compiles.
> 
> For some weird reason tho, having heavy compile jobs running now seems to block new processes from spawning. e.g. opening a new terminal window either doesn't spawn any window at all, or I won't get a shell, until I stop/pause the compile job that is. Already running shells are unaffected.

 

has been solved by switching scheduler class SCHED_IDLEPRIO. I don't know why or how SCHED_BATCH has such an effect.

I'd mark this thread as solved, but it still feels like I merely worked around a more serious problem. I wonder if anyone has some insight?Last edited by haarp on Tue Nov 12, 2019 6:51 pm; edited 1 time in total

----------

## NeddySeagoon

haarp,

```
- MAKEOPTS="-j8"

- plenty of spare RAM, no swapping taking place 

...

32G RAM ..
```

By default you have 16G max of tmpfs and MAKEOPTS="-j8" can consume 16G of RAM on large C++ projects.

tmpfs is flexible. It only takes an much RAM as it needs, up to the maximum, so if your builds wanted more that 16G, its probably OK.

It does not follow that because swap is not being used that there is no swapping happening.

swap space is used only for RAM that does not have a permanent home on disk. Dynamically allocated RAM.

The kernel has other ways to swap too.

It can drop read only data and reload it when required. That includes caches and pages of executable code too.

So the kernel can drop and reload bits of Firefox while you use it. You get lots of page faults and context switches

which do nothing for program execution but take a long time. 

The kernel can flush 'dirty' RAM to disk and drop the content.

----------

## Tony0945

 *NeddySeagoon wrote:*   

> By default you have 16G max of tmpfs and MAKEOPTS="-j8" can consume 16G of RAM on large C++ projects.
> 
> tmpfs is flexible. It only takes an much RAM as it needs, up to the maximum, so if your builds wanted more that 16G, its probably OK.

 

How to increase that maxium for those of us with 32->64 GB RAM?

----------

## NeddySeagoon

Tony0945,

The default is 50% RAM. Give the size option in fstab.  Half my 16G is no longer enough to build things. 

```
# build in RAM but 50% RAM is now too small

# and the motherboard is maxed out.

/dev/shm                 /var/tmp/portage      tmpfs noatime,nodev,nosuid,size=9G   0 0
```

size=n% works too.

For completeness, changing the size when something is building works.

The content of tmpfs is not lost.

----------

## Tony0945

Awesome, Neddy. Thanks!

----------

