Background jobs on a small server, with or without Redis

Anything that takes longer than a request should tolerate doesn't belong in the request: sending mail, resizing images, calling a slow third-party API, generating a report. The standard answer is a queue and a worker, and the standard queue needs Redis - which you may not have, may not want to pay for, and certainly do not need on day one. A database you already run will hold a queue perfectly well at the volumes a small server sees, and for a single scheduled task a cron entry beats both. This post is the ladder from "do it after the response" to "a separate worker process", where each rung is worth climbing, and what it costs when you get one wrong.

Four ways to run work outside a request#

The first one deserves a fair hearing, because it is genuinely fine sometimes:

app.post("/signup", async (req, res) => {  const user = await createUser(req.body);  res.status(201).json(user);          // answer now  sendWelcomeEmail(user).catch((error) => log.error(error));  // do this after});

The request is fast, the user is created, and the email happens afterwards. What you have given up is everything that makes a queue a queue: if the process restarts in the next 200 milliseconds the email never happens and nothing anywhere records that, there is no retry when the mail provider returns a 503, and no way to see what is pending. For a welcome email that is a shrug. For an invoice it is a bug report you will receive in three weeks with no evidence.

The rule of thumb: if you would be annoyed to learn the work silently did not happen, it needs to be written down somewhere durable before you answer the request. That is the entire argument for a queue.

What a queue actually buys#

Four things, and it is worth naming them because they are the criteria for choosing one.

• Durability. The job is on disk before the request returns. A restart, a deploy or an out-of-memory stop loses nothing.
• Retries with backoff. The mail provider is down for four minutes; the job tries again at 1s, 2s, 4s, 8s and succeeds, instead of failing once and disappearing.
• Backpressure. Ten thousand jobs arriving does not become ten thousand concurrent operations. The worker takes what it can handle.
• Visibility. You can answer "what is pending, what failed, how old is the oldest job" - which is the only monitoring of background work that means anything.

Note what is not on the list: speed. A queue makes the response fast by moving work, not by doing it faster. And every queue is at-least-once by default, which has consequences covered further down.

Queues without Redis#

Most people reach for Redis because the library they found requires it. If you already run a relational database, you already have everything a queue needs - durability, transactions, and since PostgreSQL 9.5, the one piece of SQL that makes it efficient:

-- One worker claims one job. SKIP LOCKED means other workers-- step over this row instead of queueing behind the lock.UPDATE jobsSET status = 'running', started_at = now(), attempts = attempts + 1WHERE id = (  SELECT id FROM jobs  WHERE status = 'queued' AND run_at <= now()  ORDER BY priority DESC, run_at  FOR UPDATE SKIP LOCKED  LIMIT 1)RETURNING id, kind, payload;

Without SKIP LOCKED, twenty workers polling the same table serialise behind one row lock and your queue has a throughput of one. With it, each worker takes a different job and the pattern scales to thousands of jobs a minute on hardware that is not impressive. That is far past what a small application produces.

The bigger win is transactional enqueue. If the job row is written in the same transaction as the business data, it is impossible to have a user without their welcome email job, or an email for a user whose creation rolled back. No external broker can give you that, and it removes a whole category of "how did that happen" bugs.

Libraries that do this properly, so you do not write the SQL:

The trade-offs are real and small: polling adds a little constant load (use LISTEN/NOTIFY or a one-second poll, not 50 milliseconds), the jobs table grows if you never delete completed rows, and dead rows need vacuuming - Postgres vacuum and bloat explains why a high-churn table is the classic case. Delete completed jobs after a day; keep failed ones longer.

For genuinely low volumes on one machine, SQLite works with the same pattern. Turn on WAL mode (PRAGMA journal_mode=WAL) and set a busy_timeout so concurrent writers wait instead of erroring. It is a single file with no service to run, and it caps out well above the few jobs a minute most side projects generate.

One thing to be clear about: RE:NODE does not sell Redis. Database hosting here is PostgreSQL and MongoDB, and the database slots included with app and web plans are created in the panel with a generated host, user and password. So the Postgres-backed route above is the one that works with what you can buy, and it is genuinely the right default at this size. Redis, and whether you actually need it yet makes the same argument from the other direction.

Bringing your own Redis#

Sometimes you want the Redis ecosystem anyway - BullMQ's delayed jobs and repeatable jobs are good, Celery is the default in the Python world, and an existing codebase may simply assume it. You have two honest options: rent Redis from a provider that sells it, or run it yourself on a machine you control. A VDS will do that comfortably; on RE:NODE those are prepared by hand and delivered within 24 hours rather than in a minute, so order it before the weekend you plan to use it.

If you do run your own, three settings decide whether it is a queue or a cache pretending to be one:

maxmemory-policy noevictionappendonly yesappendfsync everysec

Why each one earns its place, plus the rule that is not a line of config:

• `noeviction` is mandatory for BullMQ and a very good idea for anything else. The default eviction policies delete keys when memory fills - which, in a queue, means silently deleting jobs.
• `appendonly yes` gives you the append-only log. With snapshots alone you lose every job since the last save when the process dies.
• Bind it to localhost or a private address, and set a password. A Redis open to the internet is found within hours and is trivially used to run commands.

A minimal BullMQ worker, with the options that matter:

import { Worker, Queue } from "bullmq";const connection = { host: "127.0.0.1", port: 6379 };export const emails = new Queue("emails", { connection });new Worker("emails", async (job) => {  await sendEmail(job.data);}, {  connection,  concurrency: 2,                              // a hard ceiling, not a suggestion  removeOnComplete: { count: 1000 },           // or Redis grows forever  removeOnFail: { age: 7 * 24 * 3600 },});await emails.add("welcome", { userId: 42 }, {  attempts: 5,  backoff: { type: "exponential", delay: 1000 },  jobId: `welcome:42`,                         // dedupe: same id, one job});

removeOnComplete is the line people discover after their Redis fills up. Completed jobs are kept by default so you can inspect them, and on a small instance that becomes the memory limit in a week.

For Python, Celery needs a real broker: Redis or RabbitMQ. The database-backed transports that existed in older versions are not supported in Celery 5, so "Celery without a broker" is not a thing - if you have no broker, use procrastinate or APScheduler with a database job store instead. A Celery worker sized for a small box:

$ celery -A myapp worker --loglevel=info \    --concurrency=2 --prefetch-multiplier=1 \    --max-tasks-per-child=200 --time-limit=600 --soft-time-limit=540

--prefetch-multiplier=1 stops a worker grabbing a pile of jobs it has not started, which matters during restarts. --max-tasks-per-child recycles the child process periodically and is the cheapest available answer to a slow memory leak.

Scheduling: cron, beat, and the Schedules tab#

Queues run work when it is asked for. Something still has to ask on a timetable.

In-process schedulers - node-cron, APScheduler, setInterval - are the simplest thing that works, and they have exactly one failure mode: two processes means two runs. If you ever start a second instance, the nightly report is generated twice and emailed twice. Guard it with a lock rather than trusting yourself:

-- Returns true for exactly one caller; released when the session ends.SELECT pg_try_advisory_lock(hashtext('nightly-report'));

Celery beat is the same idea as a separate process. Run exactly one beat, ever. Two beats produce duplicate schedules, and because beat only enqueues (workers execute) the duplication is invisible until someone reads a log.

The panel Schedules tab is the option people overlook on a container-based host, where there is no system crontab to edit. It takes a cron expression and runs ordered tasks with delays between them: a console command, a backup, or a power action. On RE:NODE that is available on every plan. Two things it is very good at:

• A nightly backup followed, two minutes later, by a restart. Ordered tasks with a delay is exactly the primitive for that.
• Triggering your own job. A console command is written to your application's standard input, so a few lines of stdin handling turn the Schedules tab into a scheduler your app obeys:

process.stdin.on("data", (chunk) => {  for (const line of chunk.toString().split("\n")) {    if (line.trim() === "jobs:nightly") runNightly().catch((e) => console.error(e));  }});

Check which timezone the schedule is interpreted in before you rely on "03:00" meaning 03:00 where you live, and remember that a cron expression has five fields with no seconds - cron expressions explained covers the syntax, and scheduled tasks worth having covers which ones actually earn their place.

Sizing a worker on one box#

The constraint on a small plan is not job throughput, it is the memory and CPU the worker takes away from the thing serving users.

--concurrency=4 on a Python worker is four child processes, so a 1 GB plan running a web server and that worker is already tight. Start at concurrency 1 or 2 and raise it only when the oldest-job age says you need to.

CPU is stricter, because on RE:NODE the CPU share is a hard throttle rather than a target - a server sitting at 100% is slow, not broken, and is never suspended for it, but everything in that container shares the ceiling. A worker doing image conversion at full tilt will make your web requests slow, and no amount of process priority changes the total. Two ways out:

• Rate-limit the worker deliberately. Concurrency 1, and a small delay between jobs, is often enough to keep the site responsive while the backlog drains slightly slower.
• Put the worker on its own server. A second app plan is its own container with its own memory and CPU share, so heavy jobs cannot starve the web process. Both servers connect to the same PostgreSQL plan, which is reached on its own host and port, so the queue is shared without anything clever.

Memory is where a worker fails hardest. Reaching the limit stops the container and restarts it clean instead of swapping - so an in-flight job dies mid-write. Which brings us to the part that decides whether that matters.

Jobs that run twice, and jobs that never run#

Every queue worth using is at-least-once. A worker that claims a job and dies before acknowledging it will have that job redelivered, because the alternative - acknowledging first - loses jobs instead. Duplicates are the price of never losing work, and the way to pay it is idempotency.

• Make the operation safe to repeat. INSERT ... ON CONFLICT DO NOTHING on a unique key, a sent_at column checked before sending, a provider-side idempotency key on the payment call.
• Use a deterministic job id. BullMQ's jobId, a unique constraint on (kind, entity_id) in your own table. The same job enqueued twice becomes one job.
• Acknowledge late, and mean it. Mark the job done after the side effect, not before.
• Cap the attempts. attempts: 5 with exponential backoff, then move it to a dead-letter state where a human can look. A job retrying forever against a permanent error is a way of hiding a bug and burning CPU at the same time.
• Set a time limit. Celery has --time-limit and --soft-time-limit; BullMQ has no per-job hard timeout of its own, so wrap the handler in a Promise.race with a timer. A job that hangs on a socket with no timeout occupies a worker slot indefinitely, and that is how a queue stops moving while every dashboard says the worker is alive.

Restarts deserve their own handling. On SIGTERM, stop taking new jobs, finish the current one if it fits in the grace period, then exit - await worker.close() in BullMQ, a warm shutdown in Celery. Graceful shutdown and health checks has the full pattern, including why a long job needs checkpoints rather than a longer timeout.

What to watch#

Queue monitoring has one metric that matters and several that look like they do.

• Oldest pending job age. This is the one. If the oldest queued job is four hours old, the queue is broken regardless of what the other numbers say. Alert on it.
• Queue depth is a poor alert on its own: 5,000 jobs draining fast is fine, 12 jobs stuck forever is not.
• Failure rate per job type. One type failing is a bug; everything failing is a dependency.
• Worker heartbeat. A worker that exited quietly leaves a queue that fills silently, which is the most common background-job outage there is.

Expose these as counts from the same place your app reports health, and keep the alerting boring - see monitoring that tells you something. If the queue lives in your database, all four are a single SQL query, which is one more argument for putting it there. Just keep an eye on the connection count: a worker pool and a web pool against the same small database add up, and connection pools and limits explains what happens when they exceed what the server will accept.

FAQ#

Do I need Redis for background jobs?

No. A PostgreSQL table with SELECT ... FOR UPDATE SKIP LOCKED handles far more throughput than a small application generates, and it lets you enqueue a job in the same transaction as the data that caused it. Redis is worth adding when you want a specific library's features, not as a default.

Can I run the worker in the same process as my web app?

Yes, and on a small server it is often the right call - one process, one deploy, less to go wrong. Move it out when jobs are CPU-heavy enough to slow requests, or when you want to restart one without the other.

Why did my job run twice?

Because queues are at-least-once. A worker that crashes or is killed after starting a job but before acknowledging it causes redelivery. Make the work idempotent instead of trying to make delivery exactly-once, which no queue actually provides.

How do I run a scheduled task on a host with no crontab?

Use the panel's Schedules tab with a cron expression, which can send a console command, take a backup or restart the server. If the task belongs inside your application, have the app read a line from standard input and trigger the job when it sees it.

How many workers should I run?

Start with one, with a concurrency of one or two. Raise it only when the oldest-job age is growing, and stop when memory or the CPU share is the binding constraint. More workers than you have CPU share simply makes every job slower.

What happens to a running job if the server restarts?

With late acknowledgement, it returns to the queue and runs again - so it must be safe to repeat. Without late acknowledgement, it is lost. Test this deliberately: kill the worker mid-job and check that the outcome is one of those two and not a half-finished write.