Django Celery with Valkey: broker, beat and memory limits

Celery in Django is three pieces of wiring and four settings you should not leave at their defaults. The wiring: a celery.py next to settings.py, an import in __init__.py, and CELERY_-prefixed settings read from Django's own settings module. The settings: acks_late, the prefetch multiplier, the visibility timeout and a memory ceiling per child process. Valkey is the broker, and because it speaks the Redis protocol you point Celery at it with an ordinary redis:// URL.

The part that goes wrong on a small plan is not the configuration but the arithmetic. A Celery worker is a copy of your Django project, repeated once for the parent and once per child process, and it sits next to gunicorn, which is the same project repeated again. This post sets up the broker and the beat schedule, then spends most of its length on how many processes fit in the memory you bought and what to do when they do not. It assumes the project is already deployed as described in deploy Django to production with gunicorn, and it does not repeat the library-by-library comparison in job queues with Valkey.

What you are building#

Three kinds of process share one Valkey. The web process (gunicorn) enqueues a task and returns. One or more workers take tasks off the queue and run them. One beat process, and exactly one, adds periodic tasks to the queue on a clock. Valkey holds the messages in between.

Two things in that picture shape everything below. Beat never runs tasks, it only sends them, so it is tiny. And the worker needs its own database connections, because it imports your models and talks to the database exactly as the web process does. Connection pools and limits has the arithmetic for how those add up.

Installing Celery for Django#

Install Celery with the Redis transport extra, which pulls in the client library that also talks to Valkey:

$ pip install "celery[redis]"

Pin the version in requirements.txt like everything else. Celery's configuration key names changed in the 4.x series and several defaults moved between 5.x releases, so a range that floats across a minor version is how a restart at three in the morning changes behaviour. Python requirements and virtualenvs covers lock files.

The Django integration is three files. First the Celery application, in the same package as settings.py:

import osfrom celery import Celeryos.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings.production")app = Celery("myproject")app.config_from_object("django.conf:settings", namespace="CELERY")app.autodiscover_tasks()

Then make sure it is imported when Django starts, so that @shared_task decorators bind to this application:

from .celery import app as celery_app__all__ = ("celery_app",)

namespace="CELERY" means every Celery setting in settings.py carries a CELERY_ prefix and is upper case: the Celery setting task_acks_late becomes CELERY_TASK_ACKS_LATE. autodiscover_tasks() imports a tasks.py module from every app in INSTALLED_APPS. A task that lives anywhere else is never registered, and the worker will log Received unregistered task when a message for it arrives.

Tasks go in tasks.py inside the app, using shared_task so the app does not import the Celery instance:

from celery import shared_taskfrom .models import Order@shared_task(    autoretry_for=(ConnectionError, TimeoutError),    retry_backoff=True,    retry_backoff_max=600,    retry_jitter=True,    max_retries=5,)def send_receipt(order_id):    order = Order.objects.get(pk=order_id)    order.send_receipt_email()

Pass the primary key, not the model instance. A task carrying a whole object works with whatever it looked like when the message was written, which may be an hour ago if the queue is backed up.

The broker settings#

The connection and the behaviour settings go in the Django settings module, read from the environment so the password is not in the repository:

import osCELERY_BROKER_URL = os.environ["VALKEY_URL"]          # redis://:password@host:port/0CELERY_RESULT_BACKEND = os.environ["VALKEY_URL"].replace("/0", "/1")CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = TrueCELERY_TASK_ACKS_LATE = TrueCELERY_TASK_REJECT_ON_WORKER_LOST = TrueCELERY_WORKER_PREFETCH_MULTIPLIER = 1CELERY_TASK_TIME_LIMIT = 300CELERY_TASK_SOFT_TIME_LIMIT = 240CELERY_RESULT_EXPIRES = 3600CELERY_TASK_IGNORE_RESULT = TrueCELERY_BROKER_TRANSPORT_OPTIONS = {    "visibility_timeout": 3600,    "global_keyprefix": "shop:",}CELERY_TIMEZONE = "UTC"

Each one is there for a reason.

The URL comes from your Valkey plan's connection details: a host, a port and a generated password, none of which are the defaults of a local install. If the password contains characters such as @, / or #, percent-encode them in the URL (@ becomes %40) or the client will split the string in the wrong place and report a confusing host error. Python's urllib.parse.quote does it.

CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP matters on restart. Without it, a worker that starts a few seconds before the broker is reachable exits, and recent Celery 5.x releases print a deprecation warning telling you to set it. A hosting panel restarts everything at once after a deploy, so the order of arrival is not under your control.

The database number after the slash is a logical database inside the same Valkey. Putting results in database 1 keeps them out of the queue's keyspace, so you can inspect or flush one without touching the other. If you never read task return values, CELERY_TASK_IGNORE_RESULT = True and no result backend at all is the cheapest option and removes a whole class of leftover keys.

Acks, prefetch and the visibility timeout#

The Redis transport has no native acknowledgement, so Celery builds one. A message taken by a worker moves into a hidden structure called unacked. If the worker acknowledges within the visibility timeout the message is deleted; if it does not, the message is put back on the queue and another worker runs it.

That gives three rules.

1. The visibility timeout must be longer than your longest task, including its retries' countdowns. The default is one hour. A task that runs for ninety minutes under the default is delivered a second time while the first is still running. This is the most common cause of "my task ran twice" on a Redis-protocol broker. Notice that ETA and countdown tasks are held by the worker, not by Valkey, until due, so a task scheduled for tomorrow with a one-hour timeout is redelivered every hour. Short timeouts and long countdowns do not mix; schedule far-future work with beat instead.
2. With `acks_late`, a task may run more than once. If a worker is killed mid-task, the message returns and the task starts again from the top. Make tasks idempotent: a unique constraint on an event id, or a status column checked first. CELERY_TASK_REJECT_ON_WORKER_LOST extends this to a child killed by the kernel for memory, which is the case that matters on a small plan.
3. Prefetch 1 trades throughput for fairness. The default multiplier of 4 means each child reserves four messages in advance. If one of them is a ten-minute report, the three behind it wait, even if other children are idle. For tasks of mixed length, 1 is right; for thousands of tiny identical tasks, the default is faster.

acks_late without reject_on_worker_lost has one more trap: if the child is killed by the out-of-memory killer, the message is acknowledged as failed in some versions rather than retried. Setting both is what makes a memory kill behave like a retry.

Enqueue after the transaction commits, not before. Inside a request wrapped in a database transaction, a task enqueued with delay() can reach a worker before the row it refers to exists, and fail with DoesNotExist:

from django.db import transactiondef create_order(request):    order = Order.objects.create(...)    transaction.on_commit(lambda: send_receipt.delay(order.pk))

on_commit runs the callback once the outermost transaction commits, and discards it on rollback, which is what you want for a task that describes work caused by the data.

Beat: the periodic schedule#

Beat reads a schedule and sends tasks at the due time. It is its own process, and there must be exactly one running per schedule. Two beats means every periodic task is sent twice. This is the single most important fact about scaling Celery: you can add workers freely, but never add a second beat.

The simplest schedule lives in settings:

from celery.schedules import crontabCELERY_BEAT_SCHEDULE = {    "clear-expired-sessions": {        "task": "accounts.tasks.clear_sessions",        "schedule": crontab(hour=3, minute=30),    },    "daily-digest": {        "task": "notifications.tasks.send_digest",        "schedule": crontab(hour=7, minute=0, day_of_week="mon-fri"),        "options": {"expires": 3600},    },    "sync-stock": {        "task": "shop.tasks.sync_stock",        "schedule": 300.0,        "args": (),    },}

A bare number is an interval in seconds, measured from when beat starts. A crontab fires at wall-clock times and uses CELERY_TIMEZONE; set it explicitly, because the default is UTC and a digest "at 07:00" is otherwise not at 07:00 where you live. If cron syntax is hazy, cron expressions explained is the reference, with the caveat that Celery's crontab() takes keyword fields rather than a five-part string.

The expires option is worth copying. If workers are down for an hour, a schedule that fires every five minutes leaves twelve identical sync_stock messages waiting, and they all run in a burst when the workers return. With expires the stale ones are discarded unread.

Start beat in its own process:

$ celery -A myproject beat --loglevel=INFO -s /tmp/celerybeat-schedule

-s sets where beat keeps celerybeat-schedule, a small file recording when each entry last ran. The default writes it into the working directory. Beat is happy to lose it on restart; it then works out what is due from the clock, which for crontab entries is correct and for interval entries means "start counting again". A task that must run exactly once per day should be a crontab entry, not a 86400-second interval.

You will see advice to run beat inside the worker with celery worker -B. It saves a process, and it is fine for a single worker on a small server, but it is the reason people end up with two beats the day they add a second worker. Use it knowingly, with --concurrency set explicitly and the worker count fixed at one.

For schedules that staff should edit without a deploy, django-celery-beat stores them in the database and adds admin pages:

$ pip install django-celery-beat$ celery -A myproject beat --scheduler django_celery_beat.schedulers:DatabaseScheduler

Add "django_celery_beat" to INSTALLED_APPS and run python manage.py migrate. The trade is one more set of tables and a schedule that no longer lives in version control. For a handful of fixed jobs, keep the dictionary above.

Memory and concurrency on a small plan#

This is where Celery on a small server succeeds or is killed. Celery's default pool, prefork, starts one parent and then --concurrency child processes. The default concurrency is the number of CPU cores the process sees, which on a shared host can be the cores of the whole machine rather than your share. Set it yourself.

Every child imports your Django project, so each costs about what one gunicorn worker costs: commonly 100-200 MB for a modest project, more with heavy dependencies. How much RAM a Python app needs explains how to measure it. The parent adds a similar amount. A worker started with --concurrency 2 is therefore three processes, and with gunicorn beside it the project is loaded five or six times.

Those are starting points built from the per-process figures above, not measurements of your code. Run the worker under load and read the resident size of each process with ps -o pid,rss,cmd -C celery before you commit to a layout. On RE:NODE the console graphs memory against the plan's limit while you do it, and the limit is a kill switch: at the memory limit the kernel stops the container and it restarts clean instead of swapping. For a Celery worker that is the death of every child at once, which is exactly why acks_late and reject_on_worker_lost above are not optional.

Four measures keep a worker inside its budget.

Choose the pool by workload. Prefork is right for CPU-bound or library-heavy tasks and gives true isolation. For tasks that mostly wait on a network call, the thread pool uses one copy of the project for many concurrent tasks:

$ celery -A myproject worker --pool=threads --concurrency=8 --loglevel=INFO

Threads share the Django process, so a leaking task leaks for everyone, and the usual thread-safety caveats apply to your own code. For an email-and-webhooks workload the saving is large: eight concurrent tasks for roughly the memory of one. There is also --pool=solo, a single task at a time in the main process, which is the smallest possible footprint and a reasonable choice on a 1 GB plan with occasional jobs.

Recycle children. Python processes rarely return memory to the operating system, so a child that once handled a large export stays large. Two settings bound it:

CELERY_WORKER_MAX_TASKS_PER_CHILD = 200CELERY_WORKER_MAX_MEMORY_PER_CHILD = 300_000   # kilobytes, about 300 MB

After either limit is reached the child finishes its current task and is replaced by a fresh one. The memory limit is in kilobytes, which catches people out. Recycling is not a fix for a leak, but it turns a leak from an outage into a note in the log.

Cut the idle chatter. On a small plan, three flags remove traffic and processes you do not need when there is a single worker:

$ celery -A myproject worker --concurrency=2 -Ofair \    --without-gossip --without-mingle --without-heartbeat

-Ofair makes the parent give work only to children that are free, which matters with long tasks and prefetch above 1. --without-gossip, --without-mingle and --without-heartbeat stop the worker exchanging cluster state with other workers, useful when there are no others. Without the heartbeat, celery status and inspect commands from other nodes are less informative, so leave it on if you run several workers and rely on those.

Split queues, then limit them. A report that takes five minutes and a password reset that takes one second should not share children. Route them apart:

CELERY_TASK_DEFAULT_QUEUE = "default"CELERY_TASK_ROUTES = {    "reports.tasks.*": {"queue": "slow"},}

$ celery -A myproject worker -Q default --concurrency=2 -n fast@%h$ celery -A myproject worker -Q slow    --concurrency=1 -n slow@%h

The slow queue gets one child, so a heavy export can use at most one core's share and one process's memory, and everything urgent keeps flowing. The cost is another parent process. On a very small plan it is cheaper to keep one queue and one worker and accept that urgent tasks wait.

CPU is a separate limit from memory. On RE:NODE the CPU figure is a hard throttle to the share you bought, so a --concurrency 4 worker on half a core does not run four tasks at the speed of four cores; it runs them slowly, all at once. Concurrency above the CPU share helps only for tasks that spend their time waiting on something else.

Where the processes run#

A web process, a worker and a beat are three long-running commands. An app server on RE:NODE has one start command, so the straightforward layout is separate Python servers from the same repository, each with its own start command, all pointing at the same Valkey and database through environment variables. The web server runs gunicorn, the worker server runs celery -A myproject worker, and the beat process is added to the worker's command or given its own small server. That costs more than a single plan and removes the question of which process gets killed when memory runs out.

If you would rather keep everything in one server, a shell start command that launches the worker in the background and gunicorn in the foreground works, but be clear about what you are giving up: if the worker dies, nothing restarts it while gunicorn stays up, and the queue stops draining without any visible failure. Look at the console after every deploy, and prefer separate servers for anything that matters.

The Valkey plans start at 256 MB for $2 a month, and the Python plans start at $4. A Valkey server is reached on its own host and port with a generated password, so web and worker servers connect to it the same way from wherever they run. There is also a bundle of a Python plan, a PostgreSQL plan and a 256 MB Valkey plan, bought together at a discount on the panel shop's Bundles tab, which is the shape of this whole post: a web app, a database and a broker.

Git deploys apply to each server separately. Pushing to the repository restarts every server that has deploy-on-push switched on, so workers pick up new code on the same push as the web process. A running Celery worker keeps the old code in memory until it restarts, so this matters. A restart that arrives mid-task is a normal stop, and acks_late means the interrupted task is delivered again. Graceful shutdown and health checks covers why the stop signal should let a task finish.

Valkey settings a Celery broker needs#

Three server-side settings decide whether messages survive. Job queues with Valkey goes through them for every library; the short version for Celery:

• Eviction. Check maxmemory-policy on your instance. Anything but noeviction can delete the celery queue list or the unacked structure when memory is full. Valkey memory and eviction policies explains the policies.
• Persistence. A Valkey plan on RE:NODE saves to disk with AOF and snapshots, so queued tasks survive a restart. AOF and RDB explains what each costs.
• Size. A message is a few hundred bytes to a few kilobytes. A 256 MB instance holds a very large backlog of small messages; the way it fills is a result backend nobody expires, or large arguments.

Inspect what Celery has created with valkey-cli:

$ valkey-cli -h HOST -p PORT --askpass> KEYS *> LLEN celery> HLEN unacked> INFO memory

LLEN celery is the number of waiting messages on the default queue, and HLEN unacked is the number taken but not yet acknowledged. KEYS * is fine on a small instance and a bad habit on a large one; use SCAN there. Getting started with Valkey covers connecting.

Run celery -A myproject inspect active to see what each worker is executing right now, celery -A myproject inspect stats for pool and memory figures, and celery -A myproject purge to throw away everything waiting, which asks for confirmation and is a decision to make deliberately.

Troubleshooting#

The worker starts and then does nothing. The broker URL is wrong in a way that does not error immediately, or the tasks are on a different queue than the worker listens to. Check celery -A myproject inspect registered, then LLEN on the queue name.

`Received unregistered task of type`. The worker did not import the module that defines the task. Check autodiscover_tasks(), that the app is in INSTALLED_APPS, and that the file is named tasks.py.

`kombu.exceptions.OperationalError` or `Connection refused`. The host, port or password is wrong, or the web process has no network path to Valkey. Test the same URL with valkey-cli from the same server first.

A task runs twice. The visibility timeout is shorter than the task, or a child was killed before acknowledging. Raise the timeout, set acks_late with reject_on_worker_lost, and make the task idempotent.

Periodic tasks run twice at the same moment. Two beat processes. Check for a second server, or for -B on more than one worker.

Periodic tasks never run. Beat is not running, or it is running with a different settings module than the one that holds CELERY_BEAT_SCHEDULE. Beat logs Sending due task for each one when it works.

The server restarts every few hours. A memory kill. Compare the sum of the web workers, the parent and the children against the plan with ps, then apply the measures above. A weekly scheduled restart from the Schedules tab is a blunt but effective backstop; restart schedules that help covers the timing.

`DatabaseError: connection already closed` inside a task. A long-lived worker kept a connection that the database dropped. Call django.db.close_old_connections() at the start of long tasks, or keep CONN_MAX_AGE modest in the settings the worker uses.

Tasks pile up while workers sit idle. Often the prefetch multiplier combined with one very long task. Set it to 1 and use -Ofair.

FAQ#

Do I need Celery for a Django project?

Often not. A management command run on a schedule covers nightly cleanups and digests with no broker, and a database-backed queue covers modest volumes. Celery is worth it when tasks are frequent, need retries with backoff, and must survive a restart. Background jobs on a small server compares the options.

Can Celery use Valkey as both broker and result backend?

Yes. Use the same server with a different database number, as in the settings above, or set CELERY_TASK_IGNORE_RESULT = True and skip the backend. The Django-specific alternative is django-celery-results, which stores results in your database with CELERY_RESULT_BACKEND = "django-db".

How many Celery workers should I run?

Enough children to keep the backlog near zero at peak, and no more than the memory allows. For network-bound tasks, a thread pool of 8 to 20 on one process is cheaper than eight prefork children. For CPU-bound tasks, one child per core you actually have. The right answer on a small plan is nearly always fewer than the default.

Why does the number of keys in Valkey keep growing?

Stored results with no expiry, or unacked entries that never clear because tasks are never acknowledged. Set CELERY_RESULT_EXPIRES, ignore results you do not read, and look at HLEN unacked when a worker has been killed repeatedly.

Is it safe to run beat on every server?

No. Exactly one beat process per schedule, or every periodic task is sent once per beat. Put it on one server, or embed it with -B in a single worker that you never scale out.

What happens to queued tasks when Valkey restarts?

With persistence on, the queue is reloaded from disk and workers reconnect when CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP is set. Without persistence, every waiting task is gone. Tasks that were in unacked return to the queue after the visibility timeout.