Decrease CPU usage

[hakavlad]

There is no sense to check memory 10 times per second if there is more than one gigabyte of available memory. You can optimize the frequency of memory checks to decrease the frequency of memory checks and decrease CPU usage. Now the sleep period is 0.1. You can find the time until the next check as follows:

t1 = MemAvailable / 4000000
t2 = SwapFree / 1000000
sleep_until_next_mem_check = t1 + t2

I implemented a similar algorithm in nohang, which is OOM preventer written in Python. If nohang checked memory 10 times per second, then the CPU usage would be too large. Now when using this algorithm for calculating the period between memory checks, the CPU usage is significant only at a low memory level. If there is a lot of memory available, then nohang can use the processor even less than earlyoom. Demo https://youtu.be/8vjeolxw7Uo

I think it's possible to improve earlyoom in the same way. In most cases it will reduce the CPU usage by an order of magnitude.

[rfjakob]

I agree that could be a good idea. However, the check uses very little CPU already, so I'm not sure if it's worth it. How much CPU usage do you see from earlyoom?

[rfjakob]

The situation may be different on embedded systems though. These have much weaker CPUs.

[hakavlad]

IMHO in any case such optimization will not make it worse.

Earlyoom uses 0 - 0.7% CPU on my Pentium B960.
If I use earlyoom on VM with 1 thread, earlyoom stably uses 0.5% CPU.

[rfjakob]

There is a downside: longer reaction time

[hakavlad]

The reaction time will be sufficient to prevent OOM because the sleep period will decrease if the available memory decreases. This is proven in practice, this algorithm is used by nohang and works great.

Further I will display a nohang output when calculating sleep time as follows:
sleep_between_mem_checks = time.sleep(mem_avail / 4000000 + swap_free / 2000000)

I execute tail /dev/zero without swap and got the following nohang output

MemAvail: 4919 M, 83.7 %
sleep 1.26
MemAvail: 4919 M, 83.7 %
sleep 1.26
MemAvail: 3552 M, 60.5 %
sleep 0.91
MemAvail: 2027 M, 34.5 %
sleep 0.52
MemAvail: 1150 M, 19.6 %
sleep 0.29
MemAvail:  649 M, 11.0 %
sleep 0.17
MemAvail:  370 M,  6.3 %

2018-07-04 Wed 22:51:17
  MemAvailable (370 MiB, 6.3 %) < mem_min_sigterm (470 MiB, 8.0 %)
  SwapFree (0 MiB, 0.0 %) < swap_min_sigterm (0 MiB, - %)
  Preventing OOM: trying to send the SIGTERM signal to tail,
  Pid: 16601, Badness: 779, VmRSS: 4574 MiB, VmSwap: 0 MiB
  Success
  sleep 0.5
MemAvail: 4914 M, 83.6 %
sleep 1.26
MemAvail: 4898 M, 83.4 %
sleep 1.25
MemAvail: 4897 M, 83.3 %
sleep 1.25

Further you can see nohang output if I execute stress -m 2 --vm-bytes 4G

MemAvail: 4912 M, 83.6 %
sleep 1.26
MemAvail: 4914 M, 83.6 %
sleep 1.26
MemAvail: 2491 M, 42.4 %
sleep 0.64
MemAvail:    0 M,  0.0 %

2018-07-04 Wed 22:53:46
  MemAvailable (0 MiB, 0.0 %) < mem_min_sigkill (235 MiB, 4.0 %)
  SwapFree (0 MiB, 0.0 %) < swap_min_sigkill (0 MiB, - %)
  Preventing OOM: trying to send the SIGKILL signal to stress,
  Pid: 16675, Badness: 446, VmRSS: 2618 MiB, VmSwap: 0 MiB
  Success
  sleep 3.0
MemAvail: 4984 M, 84.8 %
sleep 1.28
MemAvail: 4978 M, 84.7 %
sleep 1.27

And stress -m 2 --vm-bytes 4G output

stress: info: [16674] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [16674] (415) <-- worker 16675 got signal 9
stress: WARN: [16674] (417) now reaping child worker processes
stress: FAIL: [16674] (451) failed run completed in 1s

OOM has been prevented without problem.

stress -m 4 --vm-bytes 4G output without swap

MemAvail: 4804 M, 81.8 %
sleep 1.23
MemAvail: 4803 M, 81.8 %
sleep 1.23
MemAvail: 1686 M, 28.7 %
sleep 0.43
MemAvail:   23 M,  0.4 %

2018-07-04 Wed 22:39:08
  MemAvailable (23 MiB, 0.4 %) < mem_min_sigkill (235 MiB, 4.0 %)
  SwapFree (0 MiB, 0.0 %) < swap_min_sigkill (0 MiB, - %)
  Preventing OOM: trying to send the SIGKILL signal to stress,
  Pid: 16536, Badness: 217, VmRSS: 1275 MiB, VmSwap: 0 MiB
  Success
  sleep 3.0
MemAvail:  116 M,  2.0 %

2018-07-04 Wed 22:39:09
  MemAvailable (116 MiB, 2.0 %) < mem_min_sigkill (235 MiB, 4.0 %)
  SwapFree (0 MiB, 0.0 %) < swap_min_sigkill (0 MiB, - %)
  Preventing OOM: trying to send the SIGKILL signal to stress,
  Pid: 16538, Badness: 290, VmRSS: 1705 MiB, VmSwap: 0 MiB
  Success
  sleep 3.0
MemAvail: 4960 M, 84.4 %
sleep 1.27
MemAvail: 4951 M, 84.3 %
sleep 1.27

and stress -m 4 --vm-bytes 4G output

stress: info: [16534] dispatching hogs: 0 cpu, 0 io, 4 vm, 0 hdd
stress: FAIL: [16534] (415) <-- worker 16536 got signal 9
stress: WARN: [16534] (417) now reaping child worker processes
stress: FAIL: [16534] (415) <-- worker 16538 got signal 9
stress: WARN: [16534] (417) now reaping child worker processes
stress: FAIL: [16534] (451) failed run completed in 3s

[hakavlad]

Nohang output when execute tail /dev/zero with swap:

MemAvail: 4873 M, 82.9 % | SwapFree: 5875 M, 100.0 %
sleep 4.26
MemAvail: 4873 M, 82.9 % | SwapFree: 5875 M, 100.0 %
sleep 4.26
MemAvail: 2063 M, 35.1 % | SwapFree: 5875 M, 100.0 %
sleep 3.54
MemAvail:    0 M,  0.0 % | SwapFree: 4524 M,  77.0 %
sleep 2.32
MemAvail:    0 M,  0.0 % | SwapFree: 2857 M,  48.6 %
sleep 1.46
MemAvail:    0 M,  0.0 % | SwapFree: 1780 M,  30.3 %
sleep 0.91
MemAvail:    0 M,  0.0 % | SwapFree: 1112 M,  18.9 %
sleep 0.57
MemAvail:    0 M,  0.0 % | SwapFree:  628 M,  10.7 %
sleep 0.32
MemAvail:    0 M,  0.0 % | SwapFree:  416 M,   7.1 %

2018-07-04 Wed 23:05:51
  MemAvailable (0 MiB, 0.0 %) < mem_min_sigterm (470 MiB, 8.0 %)
  SwapFree (416 MiB, 7.1 %) < swap_min_sigterm (470 MiB, 8.0 %)
  Preventing OOM: trying to send the SIGTERM signal to tail,
  Pid: 16775, Badness: 865, VmRSS: 5038 MiB, VmSwap: 5117 MiB
  Success
  sleep 0.5
MemAvail:  168 M,  2.9 % | SwapFree: 3271 M,  55.7 %
sleep 1.72
MemAvail: 5183 M, 88.2 % | SwapFree: 5476 M,  93.2 %
sleep 4.13

Nohang output when executestress -m 4 --vm-bytes 10G with swap:

MemAvail: 5074 M, 86.4 % | SwapFree: 5514 M,  93.9 %
sleep 4.12
MemAvail: 5070 M, 86.3 % | SwapFree: 5515 M,  93.9 %
sleep 4.12
MemAvail:    0 M,  0.0 % | SwapFree: 3793 M,  64.6 %
sleep 1.94
MemAvail:    0 M,  0.0 % | SwapFree: 1998 M,  34.0 %
sleep 1.02
MemAvail:    0 M,  0.0 % | SwapFree: 1037 M,  17.6 %
sleep 0.53
MemAvail:    0 M,  0.0 % | SwapFree:  436 M,   7.4 %

2018-07-04 Wed 23:08:05
  MemAvailable (0 MiB, 0.0 %) < mem_min_sigterm (470 MiB, 8.0 %)
  SwapFree (436 MiB, 7.4 %) < swap_min_sigterm (470 MiB, 8.0 %)
  Preventing OOM: trying to send the SIGTERM signal to stress,
  Pid: 16843, Badness: 227, VmRSS: 1279 MiB, VmSwap: 1393 MiB
  Success
  sleep 0.5
MemAvail:    0 M,  0.0 % | SwapFree:  911 M,  15.5 %
sleep 0.47
MemAvail:  805 M, 13.7 % | SwapFree: 1573 M,  26.8 %
sleep 1.01
MemAvail: 4544 M, 77.3 % | SwapFree: 5473 M,  93.2 %
sleep 3.97
MemAvail: 5151 M, 87.7 % | SwapFree: 5478 M,  93.2 %
sleep 4.12

and stress -m 4 --vm-bytes 10G output

stress: info: [16840] dispatching hogs: 0 cpu, 0 io, 4 vm, 0 hdd
stress: FAIL: [16840] (415) <-- worker 16843 got signal 15
stress: WARN: [16840] (417) now reaping child worker processes
stress: FAIL: [16840] (451) failed run completed in 9s

OOM has been totally prevented.

It is very important improvement.

As you see, 0.1s sleep period is very short for me. I want you to accept this algorithm and add CLI options for change mem/swap rates.

[rfjakob]

tail /dev/zero without swap?

[hakavlad]

With swap and without swap both.

[rfjakob]

Ah, in the other post. Sorry, missed that!

[hakavlad]

while true; do tail /dev/zero; done without swap with Nohang https://youtu.be/DefJBaKD7C8

[rfjakob]

The behavoir in your tests look very good.

However, I would like the algorithm to be easier to predict for the user. How about just dropping the poll rate to 1Hz instead of 10Hz when

(available ram + free swap) > 1gb

?

[rfjakob]

In control theory terms, your algorithm is a continous controller, and with those you usually need extra checks to keep the value from going too low or too high. So we have four values to adjust:

• swap multiplier
• ram multiplier
• lower sleep limit
• upper sleep limit

On the other hand, a two-state controller needs:

• lower sleep value
• upper sleep value
• switching threshold

But I have to admit that you could say the switching threshold are actually two values because they count ram and swap with a multiplier of one. Hmm.

Did you use an upper and lower limit in nohang?

[hakavlad]

rate_mem = 6000000
rate_swap = 3000000

mem_min_sigterm = 10 %
mem_min_sigkill = 6 %

swap_min_sigterm = 10 %
swap_min_sigkill = 6 %

No more.
See https://github.com/hakavlad/nohang/blob/master/nohang.conf
(translation coming soon)

[hakavlad]

I would like the algorithm to be easier to predict for the user.

MA/1000000 means that OOM preventer have time to prevent OOM if MemAv decreases with speed 1000000 kB/s. Is is easy to predict.

[hakavlad]

Also you can add the monitoring intensity option to be able to more rarely checks memory on systems with stable memory usage (embedded, servers).