# Limiting CPU and Memory

## Holysword

Hi there,

So, I'm sure that many have already asked about how to limit a process memory usage. As far as I could find, there is no real solution for this problem.

However, I'm running some few tests now in MATLAB and I need whatever I can have, since with no control at all MATLAB just sucks all the resources and crash my machine.

I've created a simple script which does

```
#!/bin/bash

ulimit -Sv $MEM_LIMIT

cpulimit -P /opt/matlab$VERSION/bin/glnxa64/MATLAB -l $CPU_LIMIT &

CPULIM_PID=$!

echo "cpulimit pid is $CPULIM_PID"

echo "Starting Matlab..."

/opt/matlab$VERSION/bin/matlab -desktop -nosplash

echo "Matlab has finished successfully"

echo "Cleaning cpulimit"

kill -9 $CPULIM_PID

```

First of all, Matlab NEVER finishes by itself and I have to kill it manually for the script to continue, but this is probably a Matlab issue. The second problem is that attempting to set ulimit for any non-root user makes the process to crash prematurely, without any error message, MUCH before any resource is over (I've tried to set values far above those that I actually have). The second problem is that I'm using cpulimit, which... well, its a root-only command. I've tried to symlink it to ~/bin and run it, but I am not sure if that's okay (it doesn't work anyway, because of ulimit).

I don't like the idea of running this as root. Any suggestions?

----------

## eccerr0r

What's the actual problem here?  Is matlab just eating all your RAM?

How much swap do you have?  The really crappy performance when OOM happens is typical on 0- or low-swap ("not enough swap") machines.  When the computer runs out of RAM it will do executable page swapping which is extremely slow and painful, much worse than having real swap enabled (since only executable pages can be purged, there's not that many executable pages to begin with - most of the memory is used on data pages.)  The slowness extends to your X server, and whatever you have running at the same time... making things really bad.

I suspect that matlab may be allocating pages before they're used.  With lazy allocation turned on in Linux it will let applications allocate memory before it's used, so to matlab it thinks it has all the RAM it needs.

I think the only real solutions:

- Buy more RAM

- Use a bigger swap file/partition.

----------

## Holysword

The problem is that I'm using big sparse matrices and attempting to perform ilu decomposition. If they loose sparsity, a matrix which requires 250MB can easily explode to 10GB. I need to tell matlab to stop the run if it hits such a high memory usage.

----------

## eccerr0r

How much RAM do you have?  How much swap?

Is matlab actually allocating the arrays up front (use 'top' while watching it grow)?  I no longer have access to a copy of matlab so no idea...  I suspect it may be allocating it up front with lazy allocation...

I thought Linux does honor -m (for bash's ulimit), might want to try that too.  Then again it looks like it doesn't... hum...

----------

## salahx

You might want to look into cgroups too. You can limit both memory (see Documentation/cgroups/memory.txt) and CPU usage and other things. Also you don't have to run matlab as root to put it in a cgroup (you may need root access to put it in there, but application itself doesn't need to be root)

----------

## Holysword

 *eccerr0r wrote:*   

> How much RAM do you have?  How much swap?
> 
> Is matlab actually allocating the arrays up front (use 'top' while watching it grow)?  I no longer have access to a copy of matlab so no idea...  I suspect it may be allocating it up front with lazy allocation...

 

I have matrices of 2.5GB. Some of these matrices can run an ILU decomposition nicely without any (significative) growing, while others explode to more than 100 times its original size. This is not a Matlab bug, it is a characteristic of ILU decomposition. But well, I have 4GB + 4GB swap.

 *eccerr0r wrote:*   

> I thought Linux does honor -m (for bash's ulimit), might want to try that too.  Then again it looks like it doesn't... hum...

 

Apparentely, in my machine it honors only for root.

----------

## Holysword

 *salahx wrote:*   

> You might want to look into cgroups too. You can limit both memory (see Documentation/cgroups/memory.txt) and CPU usage and other things. Also you don't have to run matlab as root to put it in a cgroup (you may need root access to put it in there, but application itself doesn't need to be root)

 

Are you talking about this? It sounds interesting, I'll take a look. Any other links you suggest?

----------

## eccerr0r

It's not a bug per se but it's the OS's (looks like linux too) capability to deal with applications that use a lot of lazy allocated pages.  The OS tries to accommodate for applications that do sparse memory allocation, but in this case, it seems to be hurting (or rather, mitigated in userland).

Again we are not talking about CPU usage.  The problem is entirely due to swap/memory, limiting CPU usage will not help.  I'm not entirely sure why RSS isn't available to the user to limit growth, probably because if there's two competing applications in memory it doesn't make sense...  In this case, having 0 swap is fine, then RSS = true allocation.

There should be a number somewhere to tell how many used pages versus allocated but unused pages, I'm surprised there does not appear to be one... weird...

Incidentally, I recall recently on an 8GB machine with 16GB swap (x86-64), I had a process using 16GB and I didn't notice way too much degradation, didn't even realize it was 8GB into swap until I checked...   It's all usage pattern of memory I suppose...

----------

## Holysword

 *eccerr0r wrote:*   

> It's not a bug per se but it's the OS's (looks like linux too) capability to deal with applications that use a lot of lazy allocated pages.  The OS tries to accommodate for applications that do sparse memory allocation, but in this case, it seems to be hurting (or rather, mitigated in userland).
> 
> Again we are not talking about CPU usage.  The problem is entirely due to swap/memory, limiting CPU usage will not help.  I'm not entirely sure why RSS isn't available to the user to limit growth, probably because if there's two competing applications in memory it doesn't make sense...  In this case, having 0 swap is fine, then RSS = true allocation.
> 
> There should be a number somewhere to tell how many used pages versus allocated but unused pages, I'm surprised there does not appear to be one... weird...
> ...

 

Just remembered that I've got a disk with 640BG that I'm not using. Someone is about to have an epic swap...

In any case, that wouldn't help the responsiveness of the system. I think everybody agrees that the ideal would be to limit both (real) memory and CPU usage, so that the process could be nearly "running in the background".

----------

## eccerr0r

Again, the problem is not CPU.  It's completely memory.

When Linux runs out of RAM and swap (whether you have 0 or gobs of it), it will swap out code pages.  Swapped out code pages, especially if it had no choice but to swap out pages that it actually needs to use frequently, will make your computer seem very slow.  Just imagine if it had no choice but to discard your keyboard interrupt driver, then every time you pressed a key it will need to refetch from disk... Even worse, mouse driver and video driver...  This is why performance tanks when you go OOM.

But as long as you have some swap and Linux does not need to do these drastic measures, it will notice you use the mouse driver and keyboard interrupt driver frequently and opt to swap out a portion of your array that it's not working on at the moment to disk.  I don't think you'll notice performance to be that bad... do you notice much of degradation when you're just 2GB into swap?

You should try the swap for the heck of it.  I have a machine with 384MB RAM and 1.5GB swap to build firefox9.  The machine swaps like mad during build, but the machine was still responsive despite being under extremely heavy swap pressure.

At least until someone knows how to get the kernel to auto-kill using RSS information!  (or you could write a script to constantly poll RSS size with ps -F and kill matlab if it gets too big...)

----------

## Holysword

 *eccerr0r wrote:*   

> Again, the problem is not CPU.  It's completely memory.

 

ILU decomposition have high cpu usage, so I still prefer to limit cpu usage as well.

 *eccerr0r wrote:*   

> do you notice much of degradation when you're just 2GB into swap?

 

Yes. I start losing responsiveness when I'm reaching full real memory usage. I don't think swap is as marvellous as you suppose. Plus, now I've got 640GB of swap, and I still lose responsiveness.

----------

## eccerr0r

Here's one thing about CPU utilization:  Doing emerge builds on Gentoo.  It also can use 100% CPU, but the machine is much more responsive than when running an application that chews up all memory.  That is, if you're also not running out of RAM during builds.

And I'm not saying you won't lose responsiveness, swap will still put heavy weight on the disk backing store and if you need to do demand fetches, swap takes priority over reading new data from disk.  If you have a slow disk this will hurt more of course (I sure hope you're not using a USB disk for swap).

If you're just going to let it sit there and crunch and occasionally touch it, yes the user interface code will still get swapped out just like the OOM situation - the unused UI code is wasting memory that it could be using for matlab, and yes it will take a few moments to swap back in.  Once it's back in however, it will be usable once more.  Keep in mind every moment you aren't using the console will make Linux think you're away and it will once more mark the UI as candidate for swapping out.

I don't "suppose" swap is good, it IS good.  It's nowhere near as good as real RAM but it does prevent the "last resort" code page swapping situation that REALLY hurts performance.  Not to mention it gives some breathing space to start new applications such as shells to kill offending processes.

Did you at least try the watchdog "solution" to write a script to monitor RSS of matlab, and kill it if it gets too big?  This will not work effectively if you're out of memory, so having enough swap is good for this too.  Also it looks cpulimit will limit child processes, so you could cpulimit a "su" to a normal user.  But this will not affect responsiveness except on 386 or really slow CPU that has a hard time dealing with only the user interface code.

----------

## Holysword

eccerr0r, I am sorry but I don't need more lessons on operational systems working, I've finished my graduation quite a long time ago, but thanks. I know what a swap is for, I am just TELLING you that it does not prevent the system from losing responsiveness - and I know why. Thanks for explaining anyway. 

So, back to where I've mentioned, the ideal would be to tell the kernel to not allocate more real memory for a given process, but only allow it to take from swap. Is that even possible with current kernels? If so, where should I start reading for allowing that?

----------

## salahx

Yes, cgroups. Here's Fedora's guide on cgroups, nad in particular, for memory cgroup you can set (eboth hard of soft limits) on amount or ram , max total ram+swap, swapiness, (though not overcommit, there was a patch years ago to that but its was never merged due to lack of a use case). Note that this is TOTAL ram - not just the RSS. THe OOM killer is also per-cgroup (by default, its on).

Current version of openrc even create a cgroup hierarchy for each controller by default in /sys/fs/cgroup. So just "mkdir /sys/fs/cgroup/mem/matlab" and set the appropriate values in the new cgroup. Then move your shell in to the new cgroup (need to be root to move your shell) and run matlab.

----------

## Holysword

 *salahx wrote:*   

> Yes, cgroups. Here's Fedora's guide on cgroups, nad in particular, for memory cgroup you can set (eboth hard of soft limits) on amount or ram , max total ram+swap, swapiness, (though not overcommit, there was a patch years ago to that but its was never merged due to lack of a use case). Note that this is TOTAL ram - not just the RSS. THe OOM killer is also per-cgroup (by default, its on).
> 
> Current version of openrc even create a cgroup hierarchy for each controller by default in /sys/fs/cgroup. So just "mkdir /sys/fs/cgroup/mem/matlab" and set the appropriate values in the new cgroup. Then move your shell in to the new cgroup (need to be root to move your shell) and run matlab.

 

Thank you salahx, I'll study that this weekend. I remember having read something regarding stability and throughput regarding cgroups. Is it stable for you?

----------

## smartass

As for sparse matrices, I've heard that NumPy handles them more efficiently....

IIRC, if matlab just allocates the memory space, VIRT will go up, but it shouldn't actually load it into physical RAM until that memory segment is used.  And it would unload it from RAM as soon as it isn't being used again if there's not much space left in RAM.

So, to me it sounds like your swap could be corrupt. Performing a HDD health test wouldn't hurt anything, would it?

----------

## Holysword

 *salahx wrote:*   

> Yes, cgroups. Here's Fedora's guide on cgroups, nad in particular, for memory cgroup you can set (eboth hard of soft limits) on amount or ram , max total ram+swap, swapiness, (though not overcommit, there was a patch years ago to that but its was never merged due to lack of a use case). Note that this is TOTAL ram - not just the RSS. THe OOM killer is also per-cgroup (by default, its on).
> 
> Current version of openrc even create a cgroup hierarchy for each controller by default in /sys/fs/cgroup. So just "mkdir /sys/fs/cgroup/mem/matlab" and set the appropriate values in the new cgroup. Then move your shell in to the new cgroup (need to be root to move your shell) and run matlab.

 

I started reading some things, and I found this:

https://forums.gentoo.org/viewtopic-t-860439-start-0.html

I am actually using jackd as well, and as so I need realtime priority for it. Cgroup + CFS are good enough for realtime applications? (Currently I'm using BFS)

----------

## Holysword

 *smartass wrote:*   

> As for sparse matrices, I've heard that NumPy handles them more efficiently....
> 
> IIRC, if matlab just allocates the memory space, VIRT will go up, but it shouldn't actually load it into physical RAM until that memory segment is used.  And it would unload it from RAM as soon as it isn't being used again if there's not much space left in RAM.
> 
> So, to me it sounds like your swap could be corrupt. Performing a HDD health test wouldn't hurt anything, would it?

 

Unfortunately in my field I'm obligated to use Matlab for comparison reason.

My swap is the second hard driver of my laptop, which I forgot when I installed my gentoo on it (so used to do everything on /dev/sda that I didn't realise there was a /dev/sdb) and was never used until now. I ran the test in any case:

```
◢ perjanus ◣ Lago $  smartctl -H -d ata /dev/sdb

smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.2.5-zen+] (local build)

Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED

◢ perjanus ◣ Lago $  smartctl -l error /dev/sdb

smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.2.5-zen+] (local build)

Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===

SMART Error Log Version: 1

No Errors Logged

◢ perjanus ◣ Lago $  
```

----------

## smartass

Oh, sorry to hear that ... just curious, what field is this ?  :Wink:  I fool around in the filed of nuclear fusion and even the JET lab switched to NumPy / SciPy. 

Well,  if your hardware is ok, that gives you at least a solid base to build on.

So, another idea:  you seem to be using a fairly recent kernel + zen patches (or is that just your naming?). I'm not saying that swap support is broken there.... but you never know.

I suggest: try running it on the oldest possible kernel that your Matlab version states in their "Requirements" as the minimum. Or just any backup kernel you still have...

Also, your sig shows that you are using the BFQ I/O scheduler. Not saying it's broken or anything, but it may not have been optimized for such huge memory mappings.

So, if older kernels don't help, try cutting down on kernel patchsets which aren't considered "production-quality" yet, especially if they have something to do with I/O.

----------

## Holysword

 *smartass wrote:*   

> Oh, sorry to hear that ... just curious, what field is this ?  I fool around in the filed of nuclear fusion and even the JET lab switched to NumPy / SciPy. 

 

Iterative solvers in general. I am using (or trying to) ILU(1e-3) as preconditioner since it seems to be the "default" for most of the test cases in most of the papers I've read. I am using test cases from University of Florida Collection, which includes problems from fluid mechanics, quantum chromodynamics, meteorology and etc. The tests are being run in Matlab because the methods I'm comparing with were published with tests coming from Matlab. I actually managed to obtain the very same results as them with their methods for the cases in which I've got already the ILU decomposition.

 *smartass wrote:*   

> Well,  if your hardware is ok, that gives you at least a solid base to build on.
> 
> So, another idea:  you seem to be using a fairly recent kernel + zen patches (or is that just your naming?). I'm not saying that swap support is broken there.... but you never know.
> 
> I suggest: try running it on the oldest possible kernel that your Matlab version states in their "Requirements" as the minimum. Or just any backup kernel you still have...
> ...

 

I am suspecting of BFS+BFQ since it is meant for regular desktops, and regular desktops do not run sparse matrices computations. Perhaps what I am seeing is an intended behaviour of BFS. Since I have to move to CFS anyway to test cgroups, I will give it a try with pure CFS.

----------

## smartass

Being able to reproduce some experiment/test data is a great feeling, isn't it ?  :Very Happy:  Here the results change each day...

Yeah, I think the more vanilla sources you use, the better. Gentoo-sources should be fine though.

----------

## Holysword

 *Holysword wrote:*   

> I will give it a try with pure CFS.

 

So, I've switched to pure CFS with 3.2.5-gentoo sources. The responsiveness seems to have increased considerably by doing so, but I think that I was having 2 problems at once. Apparentely there was a memory leak in one of the base libraries of e17 (read this post). Summarizing, CFS + updated e17 solved my problem.

Remarking here for anyone having similar issues: BFS performs really badly for non-canonical situations, like running heavy Matlab computations. In my system, it seems that the kernel was giving all the resources to Matlab, hoping it would finish sooner, resulting in frozen mouse and keyboard and pretty much everything. Even though I knew that it was Matlab's fault, I would have to sit there waiting, since I couldn't "CTRL+C" it or kill the process. CFS seems much more responsible in this sense: at first it gives a lot of resources to Matlab, and I even lose control of mouse and keyboard as well. However, after few seconds the kernel realises that "this sh** is never going to finish" and I regain control of keyboard and mouse, while Matlab runs in a backward-ish behaviour. Matlab takes more time to finish and there is normally a significative slowdown, but not a complete loss of responsiveness - that is, you can actually stop Matlab if you want.

Leaving it "unsolved" until I test the cgroups thing properly though; that would be interesting!

----------

## smartass

Glad to hear things have improved  :Smile:  Thanks for sharing this invaluable experience with BFS, I didn't realize CFS was so smart.

----------

