Graceful shutdown and health checks that mean something

A restart drops requests for one reason: your process died in the middle of serving them. The fix is two small pieces of code. One catches SIGTERM, stops accepting new connections, lets the in-flight ones finish, closes the database pool and exits with status zero. The other is an endpoint that says whether the process is ready for traffic - and, crucially, one that does not confuse "the database is having a moment" with "restart me immediately". Both take an afternoon. Together they are the difference between a deploy nobody notices and a deploy that shows up in your error tracker.

What actually stops your process#

Everything that stops a program sends it one of two things, and the distinction is the whole subject.

SIGTERM is a request. SIGKILL is not a request - the kernel removes the process and no code of yours runs. Every shutdown mechanism worth having is a race between the two: something sends SIGTERM, waits a fixed number of seconds, then sends SIGKILL. Docker's default grace period is 10 seconds. Kubernetes uses 30. Whatever is in front of your app, find out the number, because your shutdown must finish inside it or it is decoration.

Two stops skip the polite version entirely. A Kill button sends SIGKILL on purpose - that is what it is for, and it is the right thing when a process has wedged. And an out-of-memory stop gives you nothing at all: on RE:NODE, hitting the memory limit stops the container and restarts it clean rather than letting the machine swap, which is better for everyone else on the node and fatal for whatever your process was holding in memory. If losing in-flight work matters, the answer is not a cleverer shutdown handler, it is not running out of memory - Node memory limits explained covers the usual causes.

The PID 1 problem: the signal that never arrives#

Before writing any handler, check that the signal reaches your code at all. This is the most common reason a correct shutdown function never runs.

If your start command is npm start, then npm is the process that receives SIGTERM, and your application is its child. npm's behaviour with signals has varied across versions and platforms, and the failure is silent: the stop times out, everything gets killed, and your handler is never called. The same applies to a shell wrapper:

#!/bin/sh# Wrong: the shell stays as PID 1 and the signal stops herenode dist/server.js# Right: the shell replaces itself with node, which becomes PID 1exec node dist/server.js

The rules that follow from this:

• Make your start command the runtime itself: node dist/server.js, python -m gunicorn .... Not npm start, not yarn dev, not a script that forgets exec.
• If you must wrap, use exec on the last line.
• A process that is genuinely PID 1 in a container also inherits the duty of reaping orphaned children. Node and Python do not do that, which is what --init and tini are for. It matters if your app spawns subprocesses; otherwise ignore it.

Verify it rather than assuming. Log one line in the handler, stop the server from the panel, and look for that line in the console. On RE:NODE the console output is unfiltered, so a shutdown log line is actually visible rather than truncated - reading the console covers what else is worth printing there.

Draining an HTTP server in Node#

The shape is always the same: stop taking new work, finish old work, release resources, exit. In Node the subtlety is keep-alive connections, which stay open after a response and stop server.close() from ever calling back.

const server = app.listen(process.env.PORT || 3000);let shuttingDown = false;async function shutdown(signal) {  if (shuttingDown) return;  shuttingDown = true;  console.log(`${signal} received, draining`);  // 1. Fail readiness first, so anything checking stops sending traffic.  app.locals.ready = false;  // 2. Give the checker one interval to notice before we close the door.  await new Promise((resolve) => setTimeout(resolve, 5000));  // 3. Stop accepting connections; the callback fires when the last one ends.  server.close(async () => {    await pool.end();          // database    await worker?.close();     // job queue    console.log("drained, exiting");    process.exit(0);  });  // 4. Idle keep-alive sockets would hold us open forever. Node 18.2+.  server.closeIdleConnections();  // 5. A hard deadline, shorter than whatever will send SIGKILL.  setTimeout(() => {    console.error("drain timed out, forcing exit");    process.exit(1);  }, 20_000).unref();}process.on("SIGTERM", () => shutdown("SIGTERM"));process.on("SIGINT", () => shutdown("SIGINT"));

The steps that people leave out are 2 and 4.

Step 2 exists because health checks are polled, not pushed. If a load balancer checks every five seconds and you close instantly, it keeps routing requests to you for up to five seconds after you stopped listening, and every one of them is a 502. Failing readiness first, then waiting one polling interval, converts those into requests that were never sent. When nothing external is checking - a single process behind one proxy - you can shorten this, but never to zero.

Step 4 is the keep-alive trap. A browser holds an idle HTTP connection open for reuse; Node's own server.keepAliveTimeout defaults to five seconds, but a client that keeps sending requests can hold a socket indefinitely. server.closeIdleConnections() closes the ones doing nothing right now and leaves the ones mid-request alone. Node also starts marking responses on existing connections with Connection: close once the server is closing, which tells clients to go and reconnect somewhere else.

The hard deadline in step 5 must be shorter than the platform's grace period, or you have simply moved the SIGKILL later. .unref() keeps the timer from holding the event loop open when everything else has finished.

WebSocket connections never end by themselves, so server.close() will wait forever on them. Close them explicitly with a close code (1001, "going away") so clients know to reconnect, and stagger the reconnect on the client side with a random delay - otherwise every user reconnects in the same tenth of a second. WebSockets behind a reverse proxy covers the reconnect storm in more detail.

Python: gunicorn, uvicorn and friends#

Gunicorn already implements the whole pattern; the job is mostly knowing which signal does what.

--graceful-timeout defaults to 30 seconds and --timeout (how long a worker may be silent before the master kills it) also defaults to 30. If your platform's grace period is 10 seconds, gunicorn's 30 is meaningless - lower it to fit:

$ gunicorn app.wsgi:application \    --bind 0.0.0.0:8000 --workers 3 \    --timeout 30 --graceful-timeout 8 --max-requests 1000 --max-requests-jitter 100

--max-requests recycles a worker after that many requests, which is a crude but effective answer to a slow memory leak. The jitter stops every worker recycling at once.

For ASGI, uvicorn handles SIGTERM itself and supports --timeout-graceful-shutdown. In FastAPI, cleanup belongs in the lifespan handler:

from contextlib import asynccontextmanager@asynccontextmanagerasync def lifespan(app):    pool = await create_pool()    app.state.pool = pool    yield                      # the application runs here    await pool.close()         # runs on shutdown, before the process exits

Anything after yield runs during shutdown. If you are still using @app.on_event("shutdown"), it works, but lifespan is where new code goes.

Stopping a worker in the middle of a job#

An HTTP request lasts milliseconds. A background job can last minutes, and the same 10-second grace period applies, so "finish the current job" is often not an option. The answer is not a longer timeout, it is making interruption safe.

• Acknowledge late. Celery's task_acks_late=True and equivalents mean a job is only marked done after it completes. Kill the worker halfway and the job returns to the queue instead of vanishing.
• Make jobs idempotent. If late acknowledgement redelivers a job, it will sometimes run twice. Charge-the-card-twice is not a hypothetical.
• Stop consuming first. On SIGTERM, tell the worker to take no new jobs, then let the current one finish inside the deadline. Celery calls this a warm shutdown, and a second SIGTERM turns it into a cold one. BullMQ's await worker.close() waits for active jobs; passing true forces it.
• Prefetch less. A worker that has grabbed fifty jobs it has not started is holding fifty jobs hostage during a restart. worker_prefetch_multiplier = 1 in Celery, small concurrency elsewhere.
• Checkpoint long work. A job that processes 50,000 rows should record progress, so a restart resumes rather than starting again.

Background jobs on a small server goes through the queue options themselves, including what to do when you have no Redis.

Liveness, readiness and why mixing them causes outages#

Two questions, two different answers, and conflating them is how a five-second database hiccup becomes a twenty-minute outage.

• Liveness: is this process broken beyond recovery? A failure means "restart me". It should check almost nothing - that the event loop is turning and the HTTP server answers. No database, no cache, no third-party API.
• Readiness: should this process receive traffic right now? A failure means "route around me for the moment". This one may check dependencies, because starting without a database connection is a real reason not to send requests.

Put the database in the liveness check and you have built an amplifier. The database blips, every instance fails liveness, every instance is restarted, all of them reconnect at once, and the database that was briefly slow is now genuinely down. Meanwhile the restart destroyed the caches and connection pools that would have helped it recover.

A third kind is worth knowing if your app takes a while to boot: a startup check, which allows a long grace period before liveness begins. A JVM or a large Python app that needs 60 seconds to start will otherwise be killed by a liveness check with a 10-second timeout, forever, in a loop that looks exactly like a crash. If a service restarts endlessly and the logs show a clean start each time, this is usually why - the same shape of problem as a game server that keeps restarting.

A health endpoint worth having#

// Liveness: no dependencies. If this cannot answer, the process is stuck.app.get("/healthz", (req, res) => res.status(200).json({ status: "ok" }));// Readiness: cheap dependency checks, short timeouts, cached briefly.let cached = { at: 0, body: null, code: 503 };app.get("/readyz", async (req, res) => {  if (Date.now() - cached.at < 2000) return res.status(cached.code).json(cached.body);  const checks = {};  try {    await Promise.race([      pool.query("SELECT 1"),      new Promise((_, reject) => setTimeout(() => reject(new Error("timeout")), 1500)),    ]);    checks.database = "ok";  } catch (error) {    checks.database = `failed: ${error.message}`;  }  const ok = app.locals.ready && Object.values(checks).every((v) => v === "ok");  cached = {    at: Date.now(),    code: ok ? 200 : 503,    body: { status: ok ? "ok" : "degraded", checks, version: process.env.GIT_SHA },  };  if (!ok) res.set("Retry-After", "5");  res.status(cached.code).json(cached.body);});

The details that make it useful rather than decorative:

• `SELECT 1`, not a real query. You are testing that a connection works, not that the schema is correct.
• A timeout on every check. A health endpoint that hangs is worse than one that fails, because the checker's own timeout decides the outcome and it is usually much longer.
• Cache the result for a second or two. A monitor, a proxy and a deploy script all polling every five seconds is a surprising amount of load if each poll opens a connection.
• `503`, never `500`. 503 Service Unavailable with Retry-After is the correct "not now, try again"; a 500 means something threw, and monitors treat it differently.
• Return the version. Putting the commit SHA in the body turns the endpoint into a deploy check: you can see which build is actually answering.
• No secrets in the body. These endpoints end up unauthenticated. Connection strings, hostnames and stack traces do not belong there.

Health checks also generate an enormous amount of log noise - one line every few seconds, forever, drowning the lines you care about. On RE:NODE the console folds web-server noise behind a count you can switch off, which is exactly this problem. If you are rolling your own logging, exclude the health paths from the access log rather than reading past them. Logs worth keeping has the rest of that argument.

Who consumes these? Less than people assume on a small setup. Open-source nginx has passive health checking only - max_fails and fail_timeout mark an upstream bad after failed requests - and active polling is a commercial feature. So on one machine the useful consumers are your deploy script (wait for /readyz before switching traffic), your uptime monitor, and you. That is still worth having: see monitoring that tells you something for what to alert on, which is emphatically not every failed check.

Testing it before you need it#

None of this can be assumed to work. It is twenty minutes to prove:

1. Start the app and put steady load on it - autocannon -c 20 -d 30 http://localhost:3000/ or hey, anything that reports non-2xx responses.
2. Halfway through, send the real signal: kill -TERM $(pgrep -f "node dist/server.js"), or press Stop in the panel.
3. Watch the console. You should see your handler's log line, then the drain line, then the process exit.
4. Read the load tool's summary. Zero failed requests is the pass mark. A handful means the drain order is wrong; hundreds means the signal never arrived.
5. Time it. If the drain takes longer than the grace period, shorten the hard deadline until it fits.

Then break it deliberately: stop the database and check that /readyz returns 503 while /healthz still returns 200. If both fail, your liveness check is doing too much, and you have built the amplifier described above.

A weekly scheduled restart is a good forcing function for all of this. If the shutdown path is correct, a restart is invisible; if it is not, you find out on your own schedule instead of during an incident. On RE:NODE the Schedules tab runs a cron expression against ordered tasks - a power action, a backup, a console command - so a restart at 04:00 on Monday takes a minute to set up. Restart schedules that help covers when they are worth it and when they are a way of ignoring a leak.

One warning about restart loops. A crash watcher on RE:NODE polls every two minutes for uptime that went backwards; three restarts in an hour puts a warning on the server page and opens a ticket, and six suspends the server. Restarts you asked for are not counted. That is a safety net against a process that cannot start, but it also means a badly configured health check that kills a healthy app will eventually stop it altogether.

FAQ#

What is the difference between SIGTERM and SIGKILL?

SIGTERM asks the process to stop and can be caught, so your cleanup code runs. SIGKILL cannot be caught, blocked or handled - the kernel removes the process immediately. Everything that stops a container sends SIGTERM first and SIGKILL after a timeout.

How long should a graceful shutdown take?

Shorter than the grace period that will kill you anyway - typically 10 to 30 seconds. Set an explicit hard deadline a few seconds inside it and force an exit when it expires, so a stuck request cannot turn a restart into a kill.

Why does my shutdown handler never run?

Almost always because the signal went to a wrapper. If the start command is npm start or a shell script without exec, your application is a child process and never sees SIGTERM. Start the runtime directly.

Should my health check query the database?

In readiness, yes, with a short timeout. In liveness, no. A liveness check that depends on the database turns a brief database problem into a restart of every instance at once, which makes the problem worse.

What status code should an unhealthy endpoint return?

503 Service Unavailable, with a Retry-After header. Reserve 500 for genuine errors, and return 200 only when the process is actually ready to serve.

Do health checks help on a single server?

Yes, though differently. There is nothing to route around, so readiness mainly tells your deploy script when the new process is up and gives a monitor something precise to alert on. The draining half matters more: it is what makes restarts free.