5 Commands to Stop the Linux OOM Killer From Killing Apps

Your database keeps disappearing overnight, and the only clue is a kernel line saying “Out of memory: Killed process”. Here’s how to protect it using choom, OOMScoreAdjust, MemoryMax, a swap file, and systemd-oomd.

Imagine you check a server in the morning and find that MariaDB stopped at 2 AM, even though nothing changed in its configuration. The service restarts fine, the logs from MariaDB itself show nothing useful, and a few nights later it happens again.

When you finally look at the kernel log, you find that the Linux OOM killer (Out-Of-Memory killer) ended the database process because the server ran out of RAM.

What makes this confusing is that MariaDB usually wasn’t the program that used up the memory. A PHP worker, a runaway script, or a backup job asked for more memory than the server had, and the kernel picked the largest process it could find to free memory quickly, which on most servers is the database.

The usual reaction is to add more RAM or restart services on a schedule, but neither changes how the kernel chooses its victim. The good news is that Linux gives you direct control over that choice, and in this guide we will use five commands to decide which process gets killed, limit the programs that cause the problem, and give the server enough room to recover before the OOM killer is ever needed.

How the Linux OOM Killer Chooses Which Process to Kill

Before changing anything, it helps to understand how the kernel makes its decision, because every command in this guide adjusts one part of that decision.

When the kernel can’t free enough memory by dropping cache or moving pages to swap, it has to end a process to keep the whole system running. It calculates a badness score for every process, mostly based on how much RAM and swap the process is using, and it kills the process with the highest score.

You can read the current score of any process in /proc//oom_score, where a higher number means the process is more likely to be killed.

The kernel also lets you nudge that score through a second file, /proc//oom_score_adj, which accepts values from -1000 to 1000. A positive value makes a process a more attractive target, a negative value protects it, and -1000 tells the kernel never to kill that process at all.

There is one more piece that matters on modern distributions. Ubuntu 26.04 and Rocky Linux 10 both use cgroups v2 (control groups, the kernel feature that systemd uses to group and limit the processes of each service), which means you can give every service its own memory limit.

When a service reaches its own limit, the kernel runs the OOM killer only inside that service, so the rest of the server is left alone.

Confirm the OOM Killer Is Killing Your Apps

Now that you know what the OOM killer looks at, the first job is to confirm that it really is the reason your service stopped, since a crash, a failed health check, or a manual restart can look similar from the outside.

The kernel writes every OOM event to its log, so search the kernel messages in the systemd journal for both the line that starts the OOM killer and the line that names the killed process:

sudo journalctl -k -g 'oom-killer|Killed process'

Output:

Oct 21 02:14:07 tecmint kernel: php-fpm8.3 invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Oct 21 02:14:07 tecmint kernel: Out of memory: Killed process 1187 (mariadbd) total-vm:2457316kB, anon-rss:1201344kB, file-rss:0kB, shmem-rss:0kB, UID:107 pgtables:2876kB oom_score_adj:0

These two lines tell the whole story. The first one shows that php-fpm8.3 asked for memory that wasn’t available, and the second shows that the kernel killed mariadbd instead, because it was using about 1.2 GB of RAM (the anon-rss value) and had the highest score.

The -k option limits the output to kernel messages from the current boot, so if the server rebooted since the event, add -b -1 to look at the previous boot. On systems without a persistent journal, the same lines are available from the kernel ring buffer until the next reboot:

sudo dmesg -T | grep -iE 'oom-killer|killed process'

Before protecting anything, it’s useful to see which processes the kernel would pick right now. The following loop prints the OOM score, the adjustment value, and the process name for every running process, then shows the five most likely targets:

for p in /proc/[0-9]*; do printf '%s %s %s\n' "$(cat $p/oom_score 2>/dev/null)" "$(cat $p/oom_score_adj 2>/dev/null)" "$(cat $p/comm 2>/dev/null)"; done | sort -rn | head -5

Output:

668 0 mariadbd
141 0 php-fpm8.3
139 0 php-fpm8.3
42 0 node
12 0 systemd-journal

Your numbers will be different, since they depend on how much memory each process uses on your server. What matters is the order, because the process at the top of this list is the one that will be killed next, and in our case that is still the database.

If this helped you finally find out why your database keeps stopping at night, share it with someone who’s still restarting services by cron and hoping for the best.

1. Check and Change a Process’s OOM Priority with choom

Since the database sits at the top of the list, the quickest fix is to lower its priority while it’s still running, and the choom command from the util-linux package does exactly that. It’s installed by default on both Ubuntu and Rocky Linux, so there is nothing to add.

Start by checking the current values for the MariaDB process:

choom -p $(pidof mariadbd)

Output:

pid 1187's current OOM score: 668
pid 1187's current OOM score adjust value: 0

Now lower the adjustment value to -500. Lowering a value requires root, because otherwise any user could make their own processes immune to the OOM killer:

sudo choom -p $(pidof mariadbd) -n -500

Output:

pid 1187's OOM score adjust value changed from 0 to -500

If you run the first choom command again, the OOM score will be much lower, and the PHP workers or another large process will now be ahead of MariaDB in the list.

choom can also work in the other direction, which is often more useful than protecting one service. When you start a batch job, a report, or a one-off script that you can safely run again, you can raise its value so the kernel picks it first. Raising a value doesn’t need root:

choom -n 500 -- ./generate-monthly-report.sh

There is one important limitation here. The value set by choom belongs to the running process, so it’s lost as soon as MariaDB restarts or the server reboots, which is why the next command makes the change permanent.

2. Protect a Service Permanently with OOMScoreAdjust

The choom change from the previous step disappears on the next restart, so for a service managed by systemd, the right place for this setting is the service unit itself. systemd has a directive called OOMScoreAdjust that sets the same value every time the service starts.

First, check whether the MariaDB unit already sets a value, since some packages include one:

systemctl show mariadb -p OOMScoreAdjust

Output:

OOMScoreAdjust=0

Rather than editing the packaged unit file, which a package update would overwrite, create a drop-in override with systemctl edit. This command opens an editor automatically, using nano on Ubuntu and falling back to vi on minimal Rocky Linux installs where nano isn’t installed:

sudo systemctl edit mariadb

Add the following configuration in the empty area between the comment lines at the top of the file:

# /etc/systemd/system/mariadb.service.d/override.conf
[Service]
OOMScoreAdjust=-500

Here:

• [Service] tells systemd that the directives below belong to the service section of the unit.
• OOMScoreAdjust=-500 writes -500 to oom_score_adj for every process the service starts, which makes MariaDB a much less likely target without making it completely immune.

Save the file and exit the editor. systemctl command edit reloads systemd for you, but the new value is only applied when the service process starts, so restart MariaDB during a quiet period:

sudo systemctl restart mariadb

Once it’s back up, confirm that systemd and the kernel both see the new value:

systemctl show mariadb -p OOMScoreAdjust
cat /proc/$(pidof mariadbd)/oom_score_adj

Output:

OOMScoreAdjust=-500
-500

You may be tempted to use -1000 here and make the database impossible to kill. Avoid that unless you are sure the protected service can never be the one using too much memory, because if it is, the kernel will kill every other process on the server first and can still end up with nothing left to kill.

In that case, the kernel logs “Out of memory and no killable processes…” and panics, which turns one stopped service into a full server outage.

3. Cap Memory-Hungry Services with MemoryHigh and MemoryMax

Protecting the database moves the problem to the next process on the list, but the real cause is still the application that keeps growing. The better fix is to give that application its own memory limit, so that when it grows too large, the kernel deals with it inside its own cgroup and never touches the rest of the server.

This relies on cgroups v2, which Ubuntu 26.04 and Rocky Linux 10 use by default. You can confirm it quickly before setting any limits:

stat -fc %T /sys/fs/cgroup

Output:

cgroup2fs

With that confirmed, set two limits on the application service using systemctl set-property. We’ll use myapp.service as the example, so replace it with the name of your own application, such as a Gunicorn, Node.js, or PHP-FPM service:

sudo systemctl set-property myapp.service MemoryHigh=1.5G MemoryMax=2G

Here

• MemoryHigh=1.5G is a soft limit. When the service goes above it, the kernel slows the service down and reclaims its memory aggressively, which often keeps a leaking application under control without killing it.
• MemoryMax=2G is the hard limit. If the service still reaches 2 GB, the kernel runs the OOM killer only on the processes inside this service, so MariaDB and everything else keeps running.

The change takes effect immediately without restarting the service, and it stays in place after a reboot, because systemctl set-property saves it in /etc/systemd/system.control/myapp.service.d/. If you only want to test a limit until the next reboot, add the --runtime option.

You can check the active limits with systemctl show, which reports them in bytes:

systemctl show myapp -p MemoryHigh -p MemoryMax

Output:

MemoryHigh=1610612736
MemoryMax=2147483648

Pick the values by watching the service’s normal memory use first. systemctl status myapp shows the current usage on its “Memory:” line, and a sensible starting point is to set MemoryHigh a little above the usual peak and MemoryMax a bit above that.

Test a Memory Limit with a Deliberate OOM Event

Before relying on these limits, it’s worth watching one work, because the result looks very different in the logs from the global OOM event we found earlier. The following command starts a temporary service with a 200 MB limit and runs tail /dev/zero, which keeps reading data without ever finding a line ending, so its memory use grows until something stops it:

sudo systemd-run --unit=oomtest -p MemoryMax=200M -p MemorySwapMax=0 tail /dev/zero

The MemorySwapMax=0 property stops the test from pushing its memory into swap, so the limit is reached within a second or two. Now check what happened to the test service:

journalctl -u oomtest --no-pager

Output:

Oct 22 11:02:41 tecmint systemd[1]: Started oomtest.service - /usr/bin/tail /dev/zero.
Oct 22 11:02:43 tecmint systemd[1]: oomtest.service: A process of this unit has been killed by the OOM killer.
Oct 22 11:02:43 tecmint systemd[1]: oomtest.service: Main process exited, code=killed, status=9/KILL
Oct 22 11:02:43 tecmint systemd[1]: oomtest.service: Failed with result 'oom-kill'.

The kernel log confirms that this was a cgroup OOM event rather than a server-wide one, because the message starts with “Memory cgroup out of memory” instead of just “Out of memory”:

sudo journalctl -k -g 'Memory cgroup out of memory'

Output:

Oct 22 11:02:43 tecmint kernel: Memory cgroup out of memory: Killed process 4821 (tail) total-vm:207604kB, anon-rss:203796kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0

Only tail was killed, even though MariaDB is a much larger process. Once you’ve seen the result, clear the failed test unit so it doesn’t show up in systemctl --failed:

sudo systemctl reset-failed oomtest

If watching a cgroup limit kill only the misbehaving process made memory limits finally make sense, share it with a teammate who still sizes servers by adding RAM until the crashes stop.

4. Give the Kernel Breathing Room with swapon

With limits in place, the remaining risk is a short spike that no single service causes, such as a backup, a package update, and normal traffic all landing at the same time.

The OOM killer only runs when the kernel has no other way to free memory, and swap gives it another way, because it can move rarely used memory pages to disk instead of killing a process.

Many cloud images and minimal VMs ship with no swap at all, so check what the server has first:

swapon --show
free -h

If swapon --show prints nothing, the server has no swap. Ubuntu server installs usually create a /swap.img file, and Rocky Linux installs with the default LVM layout usually create a swap volume, so you only need the steps below if your server has none or very little.

Create a 2 GB swap file with dd, which writes real blocks to disk and works on both ext4 and XFS. Then lock down its permissions, since swap can contain passwords and other data from memory, format it as swap, and turn it on:

sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

If your root filesystem is Btrfs, a normal swap file won’t work, and swapon fails with “Invalid argument“. In that case create the file with sudo btrfs filesystem mkswapfile --size 2g /swapfile instead of dd, then continue from swapon.

The swap file stays active only until the next reboot, so add it to /etc/fstab. On Ubuntu and Debian, open the file with nano:

sudo nano /etc/fstab

On Rocky Linux, AlmaLinux, and RHEL, minimal installs ship with vi rather than nano, so use vi there:

sudo vi /etc/fstab

Then add the following line at the end of the file:

# /etc/fstab
/swapfile none swap defaults 0 0

Save the file and exit the editor, then confirm the swap file is active:

swapon --show

Output:

NAME      TYPE SIZE USED PRIO
/swapfile file   2G   0B   -2

Keep in mind that swap doesn’t fix a memory leak. It only gives the kernel more time, and a server that leans heavily on swap becomes very slow long before the OOM killer runs, which is exactly the situation the last command is built to handle.

5. Stop Memory Pressure Early with systemd-oomd and oomctl

The kernel OOM killer acts at the last possible moment, so a server can spend several minutes almost frozen while it swaps and reclaims memory before anything is killed.

systemd-oomd is a user-space OOM daemon that watches memory pressure instead, and it ends the right service much earlier, while the server is still responsive.

It measures pressure using PSI (Pressure Stall Information), a kernel feature that reports the percentage of time processes spend waiting for memory. Check that PSI is available on your server:

cat /proc/pressure/memory

Output:

some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

On Rocky Linux and RHEL, this command may fail with “cat: /proc/pressure/memory: Operation not supported“, because PSI (Pressure Stall Information) is disabled by default in their kernels, including RHEL 10. Enable it by adding a kernel boot argument and rebooting the server:

sudo grubby --update-kernel=ALL --args="psi=1"
sudo reboot

Next, install and start the daemon. On Ubuntu, systemd-oomd is provided as a separate package, and server installations may not include it by default.

sudo apt install systemd-oomd
sudo systemctl enable --now systemd-oomd

On Rocky Linux, AlmaLinux, and RHEL, install systemd-oomd using dnf and enable the service. The package is available in Rocky Linux 10 and AlmaLinux 10 repositories; verify availability in your enabled RHEL repositories.

sudo dnf install systemd-oomd
sudo systemctl enable --now systemd-oomd

systemd-oomd only watches the cgroups you tell it about. We’ll ask it to watch system.slice, which is the parent cgroup of every system service, so that it can pick the service causing the pressure. Open a drop-in for the slice:

sudo systemctl edit system.slice

Add the following configuration:

# /etc/systemd/system/system.slice.d/override.conf
[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=60%

• ManagedOOMMemoryPressure=kill tells systemd-oomd to monitor this slice and kill one of the services inside it when memory pressure stays too high.
• ManagedOOMMemoryPressureLimit=60% sets the memory-pressure threshold. If memory pressure exceeds 60% for longer than the configured duration, systemd-oomd may act. The default duration is controlled by DefaultMemoryPressureDurationSec in /etc/systemd/oomd.conf and is typically 30 seconds, depending on the systemd version.

Save the file and exit the editor. When systemd-oomd acts, it picks the service with the most memory reclaim activity, which is usually the one growing, but you can still keep MariaDB out of its first choices. Open the same MariaDB override we created in the second command:

sudo systemctl edit mariadb

Then add one more line under the existing OOMScoreAdjust setting:

# /etc/systemd/system/mariadb.service.d/override.conf
[Service]
OOMScoreAdjust=-500
ManagedOOMPreference=avoid

• ManagedOOMPreference=avoid tells systemd-oomd to choose other services before this one. The other accepted value, omit, removes the service from consideration entirely, and both only work for services whose cgroup is owned by root, which is the case for normal system services.

Save the file and exit the editor, then restart MariaDB so the preference is applied to its new cgroup:

sudo systemctl restart mariadb

Now oomctl shows what systemd-oomd is watching and the current pressure for each cgroup:

oomctl

The output will look similar to the following, although the exact layout depends on your systemd version:

Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 30s
System Context:
        Memory: Used: 1.4G Total: 3.8G
        Swap: Used: 0B Total: 2.0G
Swap Monitored CGroups:
Memory Pressure Monitored CGroups:
        Path: /system.slice
                Memory Pressure Limit: 60.00%
                Pressure: Avg10: 0.00 Avg60: 0.00 Avg300: 0.00 Total: 0
                Current Memory Usage: 1.1G

If /system.slice doesn’t appear under “Memory Pressure Monitored CGroups”, the drop-in wasn’t picked up, so run sudo systemctl daemon-reload and check again. When systemd-oomd does kill a service, it logs its own message instead of a kernel one, so search its journal rather than the kernel log:

journalctl -u systemd-oomd

A kill shows up as a line starting with “Killed /system.slice/” followed by the service name and the pressure that triggered it, and the affected service fails with the same ‘oom-kill‘ result we saw in the test earlier.

If systemd-oomd saved one of your servers from a slow, swapping crawl, share this guide with a colleague who has ever waited ten minutes for an SSH prompt on a server that ran out of RAM.

Pro Tips for Keeping the OOM Killer Quiet

With all five commands in place, the OOM killer now has clear instructions, and a few habits help keep it from needing them at all.

Since MemoryMax now kills only the application that misbehaves, make sure that application comes back on its own. Add Restart=on-failure and RestartSec=5 to its unit with sudo systemctl edit myapp, so a single OOM kill turns into a five-second restart instead of an outage that waits for someone to notice.

You may find advice online to set vm.overcommit_memory=2 so the kernel refuses memory requests it can’t cover. Be careful with it, because many applications reserve far more memory than they use, and with strict overcommit they start failing with “Cannot allocate memory” errors even when the server has free RAM.

It’s rarely the right answer on a general-purpose server, and the cgroup limits from the third command give you the same protection for one service at a time.

Run only one user-space OOM daemon on a server. Tools such as earlyoom do a similar job to systemd-oomd, and running both means two programs competing to kill processes under the same pressure, which makes the result hard to predict.

Finally, treat every OOM event as a signal rather than something to hide. Protecting the database keeps the site online tonight, but if journalctl -k -g 'Killed process' keeps returning new lines, the server needs either a fix for the leaking application or more memory.

Conclusion

You can now find out exactly which process the OOM killer stopped and why, protect important services with choom and OOMScoreAdjust, and contain growing applications with MemoryHigh and MemoryMax.

With a swap file for short spikes and systemd-oomd watching memory pressure, the server deals with low memory early and on your terms, instead of losing the database at 2 AM.

We’d love to hear how you’re handling memory on your own servers. Which service does the OOM killer pick most often on your systems, and did it turn out to be the real cause?

Are you using systemd-oomd, earlyoom, or only cgroup limits? Feel free to paste your journalctl -k OOM lines in the comments if you’d like help reading them.