Ubuntu

how to prevent lowmemorykiller from killing processes

Asked by Paul Gauret

Hi there,

I've upgraded a number of test systems to the latest Saucy beta. I've seen quite a few cases of processes being killed by lowmemorykiller with the following entries being logged:

[ 291.565121] lowmemorykiller: send sigkill to 4476 (python), adj 0, size 8864
[23515.006794] lowmemorykiller: send sigkill to 944 (update-apt-xapi), adj 0, size 44175

The systems where this happens are kvm virtual machines with relatively modest amounts of RAM allocated (256 or 512 MB) with 2 GB swap. The swap is mostly unused when the kills occur.

I searched around and could only find references to Android for lowmemorykiller and nothing on how to disable it. I tried the following without success:

echo '9999,9999' > /sys/module/lowmemorykiller/parameters/adj

Any ideas on how to disable lowmemorykiller or at least prevent processes to be killed rather than swapped?

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Solved by:
Paul Gauret
Solved:
Last query:
Last reply:
Revision history for this message
Manfred Hampl (m-hampl) said : 		#1

There was a similar question a few months ago
https://answers.launchpad.net/ubuntu/+question/231345

It seems to me that update-apt-xapi is a process that exceeds the capabilities of your system, and causes the OOM killer to kick also other processes. apt-xapian-index (containing update-apt-xapi) is a package to provide a searchable index on the available packages - also without that function Ubuntu should work well.

My advice: deinstall apt-xapian-index

sudo apt-get remove apt-xapian-index
or - if you want to check if there are other packages deinstalled also that you might need, test first with
apt-get --simulate remove apt-xapian-index

I know that this is not the answer to your question (it does not prevent the OOM killer to work), but it should eliminate the great majority of occurences of proccesses being killed due to update-apt-xapi overstressing your system, and leave just the very few occurences where there is a real reason.

Revision history for this message
Paul Gauret (pgauret) said : 		#2

Thanks for your advice,

I've removed the apt-xapian-index package as I didn't need it.

However I am still quite concerned by the fact that other processes are getting killed as well, with, in my view, no good reason as swap is available and should be used.

This morning again, the puppet run triggered the kills on 3 different VMs with 256 MB RAM and 2 GB of barely used swap.

[53088.588693] lowmemorykiller: send sigkill to 1462 (puppet), adj 0, size 28106

I will go ahead and create a bug report based on this information, using the lowmemorykiller may be the right thing to do on mobile devices but is definitely not appropriate on servers. At the very least there should be a documented way to disable it.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#3

When I read the comments at the top of lowmemorykiller.c source code, it seems to me that you have to provide values for both /sys/module/lowmemorykiller/parameters/adj and /sys/module/lowmemorykiller/parameters/minfree.

 * The lowmemorykiller driver lets user-space specify a set of memory thresholds
 * where processes with a range of oom_score_adj values will get killed. Specify
 * the minimum oom_score_adj values in
 * /sys/module/lowmemorykiller/parameters/adj and the number of free pages in
 * /sys/module/lowmemorykiller/parameters/minfree. Both files take a comma
 * separated list of numbers in ascending order.
 *
 * For example, write "0,8" to /sys/module/lowmemorykiller/parameters/adj and
 * "1024,4096" to /sys/module/lowmemorykiller/parameters/minfree to kill
 * processes with a oom_score_adj value of 8 or higher when the free memory
 * drops below 4096 pages and kill processes with a oom_score_adj value of 0 or
 * higher when the free memory drops below 1024 pages.

The defaults seem to be
0,1,6,12 and 1536,2048,4096,16384

So doing
echo '9999' > /sys/module/lowmemorykiller/parameters/adj
echo '1' > /sys/module/lowmemorykiller/parameters/minfree
should practically disable lowmemorykiller.

Revision history for this message
Paul Gauret (pgauret) said : 		#4

I've added entries in /etc/rc.local to change the adj and minfree settings as suggested above.

I'm still finding lowmemorykiller entries in my log files though. After investigating I found that the values in adj and minfree had somehow been reset.

# cat /sys/module/lowmemorykiller/parameters/minfree
1536,2048,4096,16384

# cat /sys/module/lowmemorykiller/parameters/adj
0,1,6,12

Couldn't find any config file in /etc that would affect them, any idea of what is resetting these values?

Revision history for this message
Manfred Hampl (m-hampl) said : 		#5

I know that it does not help solving your problem but just to make sure that setting the values works as planned, are the minfree and adj values really changing?

cat /sys/module/lowmemorykiller/parameters/minfree
cat /sys/module/lowmemorykiller/parameters/adj
echo '9999' > /sys/module/lowmemorykiller/parameters/adj
echo '1' > /sys/module/lowmemorykiller/parameters/minfree
cat /sys/module/lowmemorykiller/parameters/minfree
cat /sys/module/lowmemorykiller/parameters/adj

Revision history for this message
Paul Gauret (pgauret) said : 		#6

My bad indeed, I had forgotten to chmod +x my custom rc.local script that sets the values, so a reboot had reset them to their defaults.

With adj=9999 and minfree=1 I can indeed confirm that lowmemorykiller is disabled.

I still believe that this is a bug as the default settings are not appropriate in many cases, notably for servers. Let's see how the bug I created is received and handled.

Manfred, thanks a lot for your help and dedication to help me find a workaround, much appreciated!