Linux swap and the OOM killer: what to set and why

Swap is not extra memory. It is a place for the kernel to put pages it does not think you are using, so that the memory they occupied can be used for something else. When it works you never notice it. When the machine genuinely runs out of memory, swap does not save you; it slows down the moment at which you find out, and the process that dies is chosen by the kernel rather than by you.

Two things go wrong on small servers. The first is having no swap at all, which makes the out-of-memory killer arrive without warning. The second is having a lot of swap and a workload that touches all of its memory, which turns a fast machine into an unusable one without anything appearing to crash. This post covers how much swap to give a VDS, which knobs are real, how the OOM killer decides, and how to read the log line it leaves behind so that you fix the right thing.

What swap is actually for#

Linux divides memory in use into two kinds. Page cache is file-backed: copies of things on disk, held in memory because reading them again is free. It can be dropped at any moment, which is why free -h shows almost no free memory on a healthy server and why that is not a problem. Anonymous memory is everything else: heaps, stacks, the actual working data of your processes. It is not backed by a file, so it cannot simply be dropped. If the kernel wants to reclaim it, it needs somewhere to put it, and that somewhere is swap.

So swap buys you one specific thing: the ability to evict cold anonymous memory. That matters more than it sounds. A long-running server accumulates memory that is allocated and then never touched again - initialisation buffers, a library loaded for a feature nobody uses, a logging path taken once at startup. On a machine with no swap, every byte of that sits in RAM for months. With a small amount of swap, the kernel quietly moves it out and uses the space for page cache, and your disk reads get faster.

What swap does not buy you is capacity. If your processes genuinely need 10 GB of live working set on an 8 GB machine, swap turns a fast out-of-memory failure into a slow, grinding one. That is usually worse, because a crash is obvious and a machine running at a tenth of its speed is not.

How much swap, and how to add it#

The old rule of twice your RAM came from an era of hibernation and 256 MB machines. For a server that never hibernates, the useful amount of swap is small and roughly fixed.

Check what you already have. Many VDS images ship with none.

$ swapon --show$ free -h$ cat /proc/swaps

A swap file is as fast as a swap partition on any modern kernel and infinitely easier to resize, so use a file:

$ fallocate -l 2G /swapfile$ chmod 600 /swapfile$ mkswap /swapfile$ swapon /swapfile$ swapon --show

If fallocate produces a file the kernel refuses to use, which happens on some filesystems, write it out the slow way with dd if=/dev/zero of=/swapfile bs=1M count=2048 and repeat from mkswap. Then make it survive a reboot:

/swapfile none swap sw 0 0

Removing it later is swapoff /swapfile, then delete the file and the fstab line. swapoff has to read everything back into RAM first, so it fails if there is not enough free memory, and on a busy machine it can take a while.

Swappiness, and the knob people get backwards#

vm.swappiness is a number from 0 to 100 that sets how strongly the kernel prefers evicting anonymous pages over dropping page cache when it needs to reclaim memory. The default on nearly every distribution is 60. Since kernel 5.8 the maximum is 200, which lets you express a preference for swapping over cache reclaim rather than merely a balance.

The common misunderstanding is that it is a threshold: "swap when memory is 60% full". It is not. It is a relative preference applied only when the kernel is already reclaiming. A machine with plenty of free memory will not swap at any swappiness value.

$ cat /proc/sys/vm/swappiness60$ sysctl -w vm.swappiness=10          # until reboot

vm.swappiness = 10vm.vfs_cache_pressure = 50

Apply it with sysctl --system. 10 is the sensible value for a server running one or two latency-sensitive things: swap exists, cold pages still get evicted, but the kernel reaches for the page cache first. Setting it to 0 does not disable swap. Since kernel 3.5 it means "do not swap unless we would otherwise have to OOM", which is a legitimate setting for a database host and a bad one for a general-purpose box, because you lose the cold-page eviction that was the point.

vm.vfs_cache_pressure controls how eagerly the kernel reclaims the caches for directory entries and inodes. The default of 100 is neutral; lowering it to 50 keeps filesystem metadata in memory longer, which helps a server with a lot of small files. It is a second-order setting and you should not touch it without a reason.

The one worth knowing about and almost never worth changing is vm.overcommit_memory. The default of 0 lets the kernel accept allocations larger than the memory it has, on the reasonable assumption that most programs ask for more than they use. Setting it to 2 makes allocation strictly accounted against RAM plus swap times vm.overcommit_ratio, which makes malloc fail honestly instead of the OOM killer firing later. It also breaks a lot of software that assumes it can over-allocate, including the JVM and Redis, so leave it alone unless you have a specific reason.

The OOM killer and how it chooses#

When the kernel cannot satisfy an allocation and cannot reclaim anything, it invokes the out-of-memory killer. This is not a crash. It is a deliberate decision to terminate one process with SIGKILL so that the machine survives.

The victim is chosen by score. Every process has one, visible in /proc/<pid>/oom_score, derived mainly from how much memory it is using as a fraction of the total. The biggest consumer is usually chosen, which is why the OOM killer so reliably kills the exact thing you cared about: on a server, the largest process is your database or your game server, not the leaking cron job that pushed the machine over the edge.

You can bias that decision with /proc/<pid>/oom_score_adj, which runs from -1000 to 1000 and is added to the score. -1000 makes a process effectively exempt. 1000 volunteers it.

There are two different events that both get called OOM, and telling them apart is the single most useful diagnostic skill here:

• Global OOM. The whole machine ran out. Everything is at risk, the kernel picks a victim anywhere on the system, and the log line says constraint=CONSTRAINT_NONE and global_oom.
• cgroup OOM. One control group hit its own limit while the machine still has memory free. Only processes inside that group are candidates. The log line names a task_memcg path. This is what happens to a container, and to any systemd unit with MemoryMax= set.

That distinction decides the fix. A global OOM means buy more memory or use less. A cgroup OOM means one service exceeded a limit that you or your host set, and the rest of the machine was fine.

Reading the kill in the log#

The kernel writes a full report, and it is worth learning to read rather than skim.

$ dmesg -T | grep -iE "out of memory|killed process|oom-kill"$ journalctl -k -b | grep -i oom$ journalctl -k --since "1 hour ago" | grep -A20 "Out of memory"

The two lines that matter look roughly like this:

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,  global_oom,task_memcg=/system.slice/minecraft.service,task=java,pid=2841,uid=1000Out of memory: Killed process 2841 (java) total-vm:9412304kB, anon-rss:6182940kB,  file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13248kB oom_score_adj:0

Read anon-rss, not total-vm. total-vm is virtual address space, which for a JVM or a Go program is routinely several times the real usage and means almost nothing. anon-rss is the anonymous memory the process actually held in RAM, and that is the number to compare against your limit. In the example, a Java process holding about 5.9 GB was killed on a machine that could not find more.

Above those lines, the kernel prints a table of every process with its RSS, which tells you what else was resident at the time. That table is how you discover that the thing that died was not the thing that grew: a backup script that read a large file, or a second service that doubled in size, can push the largest process over the edge without ever being the victim.

If you find no OOM message at all and your service still disappeared, it was not the kernel. Check systemctl status for the unit's exit code, and look at why your game server keeps restarting for the other causes, which are more common than people expect.

Protecting the process you care about#

The kernel's choice can be overridden, and systemd is the clean way to do it.

[Service]OOMScoreAdjust=-500MemoryHigh=5GMemoryMax=6GMemorySwapMax=512MOOMPolicy=stop

OOMScoreAdjust biases the global kill decision so that the kernel prefers something else. MemoryHigh is a soft limit: exceed it and the cgroup is put under heavy reclaim pressure and throttled, but not killed. MemoryMax is the hard limit: exceed it and the cgroup gets its own OOM kill, leaving the rest of the machine untouched. MemorySwapMax caps how much of that limit can be swap.

The pattern worth copying is MemoryHigh a little below MemoryMax, on the services you do not trust. The soft limit gives the process a chance to reclaim before anything dies, and the hard limit means a runaway service takes itself down instead of taking the machine down. Run daemon-reload after editing, and see systemd services for your apps for the rest of the unit file.

Recent Ubuntu also ships systemd-oomd, a userspace daemon that watches pressure stall information and kills a cgroup before the kernel has to. It acts earlier and more predictably than the kernel's killer, and it logs its reasons in journalctl -u systemd-oomd. If a service is dying with no kernel OOM message on Ubuntu 22.04 or later, check there before concluding it crashed on its own.

You can watch the pressure it reacts to directly:

$ cat /proc/pressure/memorysome avg10=0.00 avg60=0.13 avg300=0.09 total=4192031full avg10=0.00 avg60=0.04 avg300=0.02 total=1104822

some is the share of time at least one task was stalled waiting for memory; full is the share where everything was. A full avg60 that is consistently above a few percent means the machine is spending real time on memory reclaim, and it is a better early warning than the free column ever was.

Swap, latency and the JVM#

Swap and garbage collection are a bad combination, and this is where most game server operators meet the problem.

A JVM's heap is anonymous memory that the collector walks periodically. If the kernel has swapped out part of that heap because it looked cold, the next major collection has to read all of it back in, page by page, through the storage device. A collection that normally takes 40 milliseconds takes several seconds. On a Minecraft server that is a visible freeze for everyone online, and it repeats. The same logic applies to any runtime with a tracing collector and to any database that keeps a large in-process cache.

Practical rules:

• Size the heap to fit in RAM with room left over. -Xmx plus the JVM's off-heap overhead plus the operating system should be comfortably under the total. Minecraft JVM flags and Java versions covers what that overhead actually is, and how much RAM a Minecraft server needs covers the sizing.
• Keep swappiness low on these machines. vm.swappiness = 10 or 1.
• Cap the unit's swap with MemorySwapMax= rather than trusting swappiness alone.
• Do not add swap to fix an undersized server. If the heap does not fit, the answer is a bigger plan, not a bigger swap file. When to upgrade your plan covers recognising the difference.

To see whether a specific process is currently swapped out:

$ grep VmSwap /proc/2841/statusVmSwap:      412360 kB$ for p in /proc/[0-9]*; do    printf "%s %s\n" "$(grep VmSwap $p/status 2>/dev/null | awk '{print $2}')" \      "$(cat $p/comm 2>/dev/null)"  done | sort -rn | head

If the process you care about has hundreds of megabytes in VmSwap, that is your latency problem, and swapoff -a && swapon -a will pull it all back into RAM immediately if you have the free memory to do it.

Thrashing: when swap makes everything worse#

Thrashing is what happens when the working set does not fit and the kernel spends its time moving the same pages in and out. The symptom is distinctive once you have seen it: the load average is high, the CPU is mostly idle, everything is slow, and nothing has crashed.

$ vmstat 1 10$ iostat -x 2          # from the sysstat package

In vmstat, the si and so columns are pages swapped in and out per second. Occasional small numbers are fine and normal. Sustained hundreds or thousands with a high wa (iowait) in the CPU columns is thrashing, and no amount of tuning fixes it. The machine needs less work or more memory.

This is also why a small swap file is a feature rather than a compromise. With 2 GB of swap, a runaway process hits the OOM killer in a few minutes and the machine recovers. With 16 GB of swap it grinds for an hour first, your monitoring alerts on latency rather than on a crash, and you spend that hour trying to SSH into a box too busy to give you a shell. Reading a server load graph has the shapes to recognise.

On a managed RE:NODE plan this whole failure mode is deliberately removed. Each server runs in its own container with a hard memory limit, and at that limit the kernel stops the container and it restarts clean rather than being left to swap. That is the opposite of upstream Pterodactyl's default, and it is a trade: you lose the chance that a brief spike gets absorbed, and you gain a server that comes back in seconds instead of one that is technically up and unusable for an hour. CPU works the same way - a hard throttle to the share bought, so a server at 100% is slow rather than broken. On a VDS, all of that is yours to configure, which is exactly what this post has been about. Choosing between a VDS and a game panel weighs the two properly.

Worth mentioning as an alternative on small boxes: zram creates a compressed block device in RAM and uses it as swap. Pages are compressed instead of written to disk, typically to a third or half their size, so you get some of the benefit of swap with none of the storage latency. On Debian and Ubuntu the zram-tools package configures it in a few lines. It is genuinely useful on a 1-2 GB machine and mostly pointless above 8 GB, where the real answer to memory pressure is to use less of it.

FAQ#

Should a server have swap at all?

Yes, a small amount. Zero swap means cold anonymous pages sit in RAM forever and the OOM killer arrives with no warning at all. One to two gigabytes gives the kernel somewhere to put pages nobody is touching, without providing enough room to thrash for an hour.

Does more swap prevent the OOM killer?

Only by postponing it. The killer fires when the kernel cannot reclaim anything, and a huge swap file means it takes much longer to reach that point, spent at a fraction of the machine's speed. If you are hitting OOM regularly, the fix is less memory usage or more RAM.

What does swappiness 0 do?

It tells the kernel to avoid swapping anonymous memory until the alternative is an OOM kill. It does not disable swap, and it does not disable page cache reclaim. Use 10 for a general server; 0 or 1 only on a host dedicated to a database or a JVM.

Why did the OOM killer kill my database instead of the thing that leaked?

Because it scores by memory usage, and your database was the largest process on the machine. The leak pushed the system over the edge; the score picked the victim. Use OOMScoreAdjust on the service you want protected and MemoryMax on the one you do not trust.

How do I tell whether a container or the whole machine ran out?

Read the oom-kill: line in dmesg. constraint=CONSTRAINT_NONE with global_oom means the machine. A task_memcg path pointing at a specific slice or container, without global_oom, means that cgroup hit its own limit while the host was fine.

Is swap on NVMe fast enough to not matter?

It is much better than it was on spinning disks, and still roughly a thousand times slower than RAM. NVMe changes swap from catastrophic to merely bad. It does not make swap a substitute for memory, and it does not save a garbage collector that has to fault its whole heap back in.