systemd services for your apps: unit files and journalctl

An application started from an SSH session dies with the session. An application started with nohup ... & survives the session and dies with the reboot. An application inside screen or tmux survives both until the day somebody kills the wrong window, and it leaves you with no logs anybody else can find. A systemd unit is twelve lines of text that fixes all three at once: the process starts at boot, restarts when it crashes, runs as the right user with the right environment, and sends everything it prints to a log you can query by time and severity.

This is the part of running your own server that pays back fastest, and it is genuinely small. Below is a working unit file, then every directive worth knowing in it, then the two things that actually go wrong - the start limit and the environment file - followed by logs, hardening, several instances of one service, and timers as a better cron.

What systemd gives you that tmux and nohup do not#

• It starts at boot, before anyone logs in, in a defined order relative to the network and the filesystems it needs.
• It restarts on failure, with a delay you set and a limit so that a permanently broken service does not spin.
• It owns the whole process tree. Everything the service spawns lives in one control group, so stopping the service stops the children too rather than leaving orphans holding a port.
• It runs as a chosen user with a chosen working directory, umask and environment, none of which depend on whose shell started it.
• It centralises output. Anything written to standard output or standard error lands in the journal, tagged with the unit, queryable by time, and rotated without you writing a logrotate file.
• It can drop privileges and restrict the kernel surface with a dozen one-line options, which is the cheapest hardening available anywhere.

What it does not do is know whether your application is working. A wedged process that no longer answers requests satisfies systemd perfectly. Liveness is a health check plus something that acts on it, which is a different job - graceful shutdown and health checks covers the application half.

Your first unit file#

[Unit]Description=Notes web applicationDocumentation=https://example.com/docsAfter=network-online.target postgresql.serviceWants=network-online.target[Service]Type=simpleUser=notesGroup=notesWorkingDirectory=/srv/notesEnvironmentFile=-/etc/notes/notes.envEnvironment=NODE_ENV=productionExecStart=/usr/bin/node /srv/notes/server.jsRestart=on-failureRestartSec=5TimeoutStopSec=30[Install]WantedBy=multi-user.target

$ sudo systemctl daemon-reload$ sudo systemctl enable --now notes$ systemctl status notes$ journalctl -u notes -f

daemon-reload after every edit, always - systemd caches unit files and will otherwise run the old one while you read the new one. enable creates the symlink that starts it at boot; --now also starts it immediately. Those two are separate concepts and systemctl start without enable is the classic reason a service that "works" is missing after a reboot.

Three details in that file that are easy to get wrong. ExecStart must be an absolute path - node will not be found, /usr/bin/node will. It is not run through a shell, so pipes, globs, && and variable expansion do not work; if you need them, wrap the command in /bin/sh -c '...' deliberately. And units placed in /etc/systemd/system/ override same-named units shipped by packages in /usr/lib/systemd/system/, which is why your own services belong in the first directory and package units should be adjusted with a drop-in rather than edited.

That drop-in mechanism is worth learning on day one:

$ sudo systemctl edit notes      # writes /etc/systemd/system/notes.service.d/override.conf$ systemctl cat notes            # shows the unit plus every drop-in, merged

The directives that matter#

The ordering pair is the one people misunderstand. After=postgresql.service says "if both are being started, start me second". It does not start PostgreSQL. Wants=postgresql.service starts it and carries on if it fails; Requires= starts it and refuses to start if it fails. Use Wants plus After for most dependencies and keep Requires for the cases where running without the other unit is genuinely pointless.

After=network-online.target deserves a note as well, because network.target is not what most people want. network.target means the network stack is being brought up; network-online.target means an address has actually been configured, and it only works if you also list it in Wants= and the distribution's wait-online service is enabled. If your service binds a specific public address at startup, you need the second. If it binds 0.0.0.0, the first is fine.

KillSignal=SIGINT looks obscure until you host a game. Several game servers save their world on SIGINT and simply die on SIGTERM, so a unit with the default kill signal loses everything since the last autosave on every restart. Check what your software expects before you rely on the restart button.

Type, Restart and the start limit#

Type= tells systemd when the service counts as started, which decides when dependent units are allowed to begin.

If you wrote the application, run it in the foreground and use simple or exec. Do not add a daemonising flag so you can use forking; it is more moving parts for no gain, and modern process managers all expect the foreground.

Restart= has more values than people use, and two of them cover almost everything:

on-failure for anything that could legitimately finish; always for a server that should never exit at all. Set RestartSec=5 alongside it, because the default of 100 milliseconds turns a crash loop into a busy loop.

Then the trap. systemd refuses to restart a service more than StartLimitBurst times within StartLimitIntervalSec - by default five starts in ten seconds - and when it hits that limit it stops trying entirely and reports:

notes.service: Start request repeated too quickly.notes.service: Failed with result 'start-limit-hit'.

This is systemd doing the right thing: a service that dies instantly five times running is broken, not unlucky. What surprises people is that the service then stays dead until a human intervenes, even after the underlying problem is fixed. The fix is sudo systemctl reset-failed notes followed by a start, and the tuning is in [Unit]:

[Unit]StartLimitIntervalSec=60StartLimitBurst=5

Widening that window rather than raising the burst is usually right. If your service takes ten seconds to fail, five failures already spans fifty seconds, and the default interval never triggers - which is why the message appears mostly with services that fail on a missing file in under a second. The same pattern in a managed context, and what a host does about it, is in why your game server keeps restarting.

Environment variables and secrets#

There are three ways to get configuration into a unit, and they layer.

Environment=NODE_ENV=productionEnvironment="GREETING=hello there"EnvironmentFile=-/etc/notes/notes.env

Environment= is for non-secret values you are happy to have in a world-readable unit file. EnvironmentFile= points at a file of KEY=value lines, and that is where secrets belong, because the file can be owned by root and mode 0600 while the unit stays readable.

$ sudo install -d -m 0750 -o root -g notes /etc/notes$ sudo install -m 0640 -o root -g notes /dev/null /etc/notes/notes.env$ sudo nano /etc/notes/notes.env

DATABASE_URL=postgres://notes:secret@127.0.0.1:5432/notesSESSION_SECRET=a-long-random-stringPORT=3000

Four rules for that file, all of which people break once:

• No `export`. It is not a shell script. export FOO=bar sets a variable literally called export FOO.
• No shell expansion. PATH=$PATH:/opt/bin stores the four characters $PATH, not your path.
• Quotes are stripped, so FOO="bar" gives bar. That means a value that genuinely needs quotes needs care.
• The leading `-` on EnvironmentFile=-/etc/notes/notes.env makes a missing file non-fatal. Without it, a typo in the path stops the service from starting at all - which is sometimes exactly what you want for a file holding the database password.

Check what the service actually received rather than assuming:

$ systemctl show notes -p Environment$ sudo cat /proc/$(systemctl show notes -p MainPID --value)/environ | tr '\0' '\n'

That second command is also the reason to think about who can read the process environment on a shared box. Environment variables and secrets covers the wider question, including what should not be an environment variable at all.

journalctl: where the output went#

Anything the service writes to standard output or standard error goes to the journal, tagged with the unit name. No redirection, no log file, no logrotate configuration.

$ journalctl -u notes                      # everything, oldest first$ journalctl -u notes -f                   # follow, like tail -f$ journalctl -u notes -n 200 --no-pager    # last 200 lines, printable$ journalctl -u notes --since "1 hour ago"$ journalctl -u notes --since "2026-09-20 18:00" --until "2026-09-20 19:00"$ journalctl -u notes -p err               # errors and worse only$ journalctl -u notes -b                   # this boot only$ journalctl -u notes -o cat               # message text, no timestamps

The -p filter takes the syslog severities, so -p warning gives warnings and above. -b -1 gives the previous boot, which is how you find out what happened immediately before an unplanned reboot.

One thing to check on a fresh server: whether the journal survives a reboot at all. On several distributions it is stored in /run and vanishes when the machine restarts, which makes post-mortem debugging impossible.

$ journalctl --disk-usage$ sudo mkdir -p /var/log/journal$ sudo systemd-tmpfiles --create --prefix /var/log/journal$ sudo systemctl restart systemd-journald

Then cap it, because an unbounded journal on a chatty service is a slow way to fill a disk:

Storage=persistentSystemMaxUse=500MMaxRetentionSec=1month

sudo journalctl --vacuum-time=14d prunes an existing journal immediately. And if you see Suppressed N messages from notes.service in the log, that is journald's rate limiter - by default a few thousand messages in a short window - which means your service is far too noisy rather than that journald is broken. Logs worth keeping is about deciding what should be in there in the first place.

Locking the service down#

Each of these is one line and most cost nothing. Add them a few at a time and test, because a couple of them do break real software.

NoNewPrivileges=yesPrivateTmp=yesProtectSystem=strictProtectHome=yesReadWritePaths=/srv/notes/uploadsProtectKernelTunables=yesProtectKernelModules=yesProtectControlGroups=yesRestrictAddressFamilies=AF_INET AF_INET6 AF_UNIXLockPersonality=yes

ProtectSystem=strict mounts the entire filesystem read-only for this service except /dev, /proc and /sys, so every directory it legitimately writes to has to be listed in ReadWritePaths=. That is the single most valuable line here and the one most likely to produce a startup failure the first time, which is the point: it forces you to know what your application writes.

Two more worth knowing. DynamicUser=yes creates and destroys a system user around the service so you never manage one, and it pairs with StateDirectory=notes, which creates /var/lib/notes with the right ownership automatically. And AmbientCapabilities=CAP_NET_BIND_SERVICE lets an unprivileged service bind port 80 or 443 without running as root - though behind an nginx reverse proxy you do not need it at all.

The one to be careful with is MemoryDenyWriteExecute=yes, which breaks any runtime with a just-in-time compiler. Node, modern Java and several Python extensions will refuse to start. Measure the result of whatever you choose:

$ systemd-analyze security notes.service

That prints a line per setting and an overall exposure score. Treat it as a checklist rather than a target - going from "unsafe" to "medium" is most of the available benefit, and the last few points usually cost more than they return.

Template units and timers#

A template unit runs many copies of one configuration. The file name ends in @, and %i is replaced by whatever follows the @ when you start it.

[Unit]Description=Valheim server (%i)After=network-online.targetWants=network-online.target[Service]Type=simpleUser=gamesWorkingDirectory=/srv/valheim/%iEnvironmentFile=/etc/valheim/%i.envExecStart=/srv/valheim/%i/valheim_server.x86_64 -nographics -batchmodeKillSignal=SIGINTTimeoutStopSec=90Restart=on-failureRestartSec=10[Install]WantedBy=multi-user.target

$ sudo systemctl enable --now valheim@midgard$ sudo systemctl enable --now valheim@testworld$ journalctl -u valheim@midgard -f

Two servers, one unit file, separate environment files, separate directories, separate logs. KillSignal=SIGINT and the long TimeoutStopSec are there because that is a game server that saves on shutdown and needs to be allowed to finish. Several game servers on one VDS is the full version of this pattern, including ports and users.

Timers replace cron, and the case for switching is not aesthetic. A timer is two files - a oneshot service and a timer that triggers it - and the service gets everything above: a user, an environment file, resource limits, hardening, and output in the journal instead of an email nobody reads.

[Unit]Description=Nightly database dump[Service]Type=oneshotUser=postgresExecStart=/usr/local/bin/db-backup.sh

[Unit]Description=Run the database dump at 04:30[Timer]OnCalendar=*-*-* 04:30:00Persistent=trueRandomizedDelaySec=300[Install]WantedBy=timers.target

$ sudo systemctl enable --now db-backup.timer$ systemctl list-timers --all$ systemd-analyze calendar "*-*-* 04:30:00"$ sudo systemctl start db-backup.service     # run it now, on demand

Persistent=true is the feature cron does not have: if the machine was off at 04:30, the job runs once shortly after it comes back rather than being silently skipped. RandomizedDelaySec spreads several machines so they do not all hit the same backup target at once. And systemd-analyze calendar tells you the next time an expression will fire, which beats waiting until tomorrow to find out you wrote it wrong. If you prefer cron's syntax, cron expressions explained covers the five fields - and note that a timer is only as good as the script it runs, so read backups that actually restore before trusting either.

When it will not start#

$ systemctl status notes          # state, last lines, exit code$ journalctl -u notes -n 50       # the actual error$ systemctl cat notes             # the unit as systemd sees it, plus drop-ins$ systemd-analyze verify /etc/systemd/system/notes.service$ systemctl list-units --failed   # everything broken on this box

`status=203/EXEC`. systemd could not execute ExecStart. Wrong path, not executable, or a script with a missing or wrong shebang. Run the exact command by hand as the service user.

`status=200/CHDIR`. WorkingDirectory does not exist, or the service user cannot enter it.

`Start request repeated too quickly`. The start limit. Fix the real problem, then systemctl reset-failed.

It starts by hand and fails at boot. A dependency was not ready. Add After= for the thing it needs, and Wants=network-online.target if it binds a specific address.

It runs but cannot write anything. ProtectSystem=strict without the matching ReadWritePaths=, or plain file ownership. The journal will name the path.

Environment variables are empty. The file path is wrong and you used the - prefix, so the failure is silent. Check with systemctl show notes -p Environment.

It works in a shell and not as a service. Almost always the environment: your login shell has a PATH, a HOME, a language setting and possibly a version manager that the service does not. Set what you need explicitly in the unit rather than inheriting it by luck.

Changes to the unit do nothing. You forgot daemon-reload.

On RE:NODE's managed plans none of this is yours to write: the panel starts and stops the process, a watcher polls every two minutes for a server that went offline or whose uptime went backwards, and three unrequested restarts in an hour raise a warning and open a ticket automatically. The Schedules tab takes a cron expression and runs ordered tasks - a console command, a backup, a power action - which is the managed equivalent of the timer above. On a VDS you have root and you build the same behaviour yourself out of the pieces in this post, which is more work and considerably more control. Choosing between a VDS and a game panel is the honest comparison, and the first hour on a new VDS is what to do before any of it.

FAQ#

Do I still need pm2 or supervisor?

Not for keeping a process alive - systemd does that better, at a lower level, with logs and dependencies included. Process managers still earn their place for features systemd does not have, such as running several instances of a Node app across cores with zero-downtime reloads. Running both is a common and avoidable mistake; pm2 or a hosting panel goes through the overlap.

Should I run services as root?

No. Create a system user with no login shell, set User= in the unit, and give it write access only to the directories it needs. If the service must bind a low port, use AmbientCapabilities=CAP_NET_BIND_SERVICE or put a proxy in front of it rather than reaching for root.

What is the difference between systemctl start and systemctl enable?

start runs it now. enable makes it start at boot. They are independent, and a service you started but never enabled is missing after the next reboot. enable --now does both.

Can I use systemd to run Docker containers?

Yes, and it is a reasonable pattern when a container needs to start after a mounted filesystem or in a specific order. Write a unit that runs docker compose up -d as Type=oneshot with RemainAfterExit=yes, or use Docker's own restart policies for the simple case - Docker on a VDS covers those.

Why do my logs disappear after a reboot?

The journal is volatile on some distributions, stored under /run. Create /var/log/journal, set Storage=persistent in journald.conf, restart systemd-journald, and cap the size so it cannot fill the disk.

How do I run a service as my own user without root?

systemctl --user manages units in ~/.config/systemd/user/. By default those stop when you log out, so run loginctl enable-linger yourname once to keep them running. It is convenient for personal tools and not the right place for anything the machine's uptime depends on.