Laravel queues with Valkey: workers, retries, failed jobs

Laravel talks to Valkey through its redis queue driver, so the setup is four lines in .env and one long-running process: php artisan queue:work redis. The driver, the retry rules and the failed-jobs table are the easy half. The half that costs people jobs is the interaction between three numbers - the job's timeout, the connection's retry_after, and how long your worker process actually stays alive - and the question of what restarts that worker when it dies, which on a single container has no obvious answer. This post covers all of it, in the order you will need it.

It deliberately does not repeat two earlier posts. Deploy Laravel to production covers the base deploy and a first queue:work with the database driver. Job queues with Valkey compares BullMQ, Celery, RQ and Laravel and explains the server settings a queue depends on. Here is the Laravel-specific depth: configuration keys, retry behaviour, failed-job handling and process supervision.

Why move the queue from the database to Valkey#

The database driver is a fine default and the right one for a small site: the job is written in the same transaction as the data that caused it, nothing extra runs, and you can inspect the queue with SQL. It starts to hurt for three reasons. Workers poll, so every idle worker runs a query every few seconds. Many workers contend for the same rows. And the jobs table shares disk and connections with the rest of your application.

Valkey takes the job traffic away from your relational database. A push is a single list operation, a worker waiting for work holds a blocking connection instead of polling, and a backlog of thousands of jobs costs memory rather than table bloat. What you give up is the transactional guarantee: with the redis driver a job can be pushed before the database transaction that created its data has committed, and a fast worker will then run it against rows that do not exist yet. Section "Dispatching inside transactions" below deals with that, and it is the first thing to check if a job fails only some of the time.

Choose Valkey when you already have it for cache or sessions, when job volume is high enough that polling shows up in your database load, or when you want Horizon. Stay on database when volume is low and correctness around transactions matters more than throughput. Background jobs on a small server argues the case for the simple option honestly.

Configure the redis queue driver#

Laravel's Redis support is used unchanged with Valkey, because Valkey speaks the same protocol. You need a client library first. Laravel supports the phpredis PHP extension and the pure-PHP predis package, selected by REDIS_CLIENT. Check what your PHP has:

$ php -m | grep -i redis$ composer require predis/predis

If php -m lists redis, use REDIS_CLIENT=phpredis. If it does not, predis needs no extension at all and works the same way, a little slower. Then set the connection details in .env:

QUEUE_CONNECTION=redisREDIS_CLIENT=phpredisREDIS_HOST=203.0.113.20REDIS_PORT=6380REDIS_PASSWORD=a-long-generated-passwordREDIS_PREFIX=shop-

REDIS_HOST, REDIS_PORT and REDIS_PASSWORD are the values your Valkey provider gives you. You can pass a single REDIS_URL instead, in the form redis://:password@host:port; it takes precedence over the separate keys. REDIS_PREFIX is worth setting by hand. By default Laravel builds a prefix from APP_NAME, which means renaming the app changes the prefix and orphans every job waiting under the old one.

The queue driver itself is configured in config/queue.php:

'redis' => [    'driver' => 'redis',    'connection' => env('REDIS_QUEUE_CONNECTION', 'default'),    'queue' => env('REDIS_QUEUE', 'default'),    'retry_after' => (int) env('REDIS_QUEUE_RETRY_AFTER', 90),    'block_for' => null,    'after_commit' => false,],

Two of those deserve a decision. block_for set to 5 makes an idle worker wait on a blocking pop for up to five seconds instead of polling in a loop, which cuts latency and Valkey traffic. Leave it null only if you have a reason. And after_commit is the transaction fix described later.

Confirm it works end to end before writing real jobs:

$ php artisan tinker --execute="dispatch(fn () => logger('queue ok'));"$ php artisan queue:work redis --once$ tail -n 3 storage/logs/laravel.log

--once processes a single job and exits, which is the right way to test. If the log line appears, your driver, credentials and prefix are right.

What Laravel stores in Valkey#

It helps to know what is in the database when something looks wrong. For each queue name, Laravel keeps three structures under your prefix:

A worker pops from the list and, in the same atomic step, adds the job to the reserved set with a score of now plus retry_after. When the job finishes it is removed from the reserved set. A job whose score has passed is moved back onto the list by the next worker that looks. That single mechanism is the whole retry-after behaviour, and it is why the next section matters.

You can look at it directly. The --scan form avoids guessing the prefix:

$ valkey-cli -h 203.0.113.20 -p 6380 --askpass --scan --pattern '*queues:*'$ valkey-cli -h 203.0.113.20 -p 6380 --askpass LLEN shop-queues:default$ valkey-cli -h 203.0.113.20 -p 6380 --askpass ZCARD shop-queues:default:reserved

A growing reserved count with an empty list means workers are taking jobs and not finishing them. A growing list with a flat reserved count means nothing is working at all. If you run Laravel on a different REDIS_PREFIX from the one you inspect, you will see an empty database and conclude the queue is broken. It is not.

Timeouts and retry_after: the bug that runs jobs twice#

Every Laravel queue bug report that says "my job ran twice" is one of two things. Either the job really failed halfway and was retried, which is correct behaviour for at-least-once delivery, or retry_after is shorter than the job. The second is a configuration error.

There are three time limits and they must nest correctly:

1. The job's own limit: the $timeout property on the job class, or --timeout on the worker (default 60 seconds). The worker kills a job that runs longer, using the pcntl extension.
2. The connection's retry_after (default 90 seconds). After this long a reserved job is assumed lost and is pushed back for another worker.
3. The worker's --max-time, after which it exits between jobs.

The rule is that the job timeout must be shorter than retry_after, by a margin. With the defaults that is 60 against 90, which is fine. If you raise a job's $timeout to 300 for a report export and leave retry_after at 90, a second worker picks up the same job at 90 seconds while the first is still running. Both finish. The customer gets two reports.

class BuildExport implements ShouldQueue{    use Queueable;    public int $timeout = 300;    public bool $failOnTimeout = true;    // config/queue.php retry_after must be larger than 300, say 330}

Set REDIS_QUEUE_RETRY_AFTER=330 in .env to match. $failOnTimeout = true marks the job failed when it times out instead of retrying it, which is usually what you want for a job that already ran for five minutes.

The wasteful alternative is one giant retry_after for everything, say an hour. It is safe against duplicates and terrible at recovering: if a worker is killed mid-job, that job sits invisible in the reserved set for an hour before anyone retries it. A better design is to keep retry_after modest and put the genuinely slow jobs on their own queue with their own worker, started with a longer --timeout, while retry_after is set above that timeout. Two connections in config/queue.php, redis and redis-long, each with its own retry_after, is a legitimate way to do exactly that.

Workers also need the pcntl PHP extension to enforce timeouts at all. If a job hangs forever and --timeout does nothing, check php -m | grep pcntl.

Tries, backoff and middleware#

How many times a job is attempted, and how long to wait between attempts, can be set on the worker or on the job. The job wins when both are present.

class SendInvoice implements ShouldQueue{    use Queueable;    public int $tries = 5;    public int $maxExceptions = 3;    public function backoff(): array    {        return [10, 60, 300, 900];    }    public function handle(Mailer $mailer): void    {        $mailer->send($this->invoiceId);    }}

The attributes that matter:

A few points that trip people up. --tries=0 means unlimited, which on a job that always fails is an infinite loop that never reaches failed_jobs. A job released with $this->release(30) counts as an attempt. $tries and retryUntil() together mean whichever limit is hit first. And a job that is retried because its worker was killed counts an attempt even though your code never raised an exception, which is why $maxExceptions exists.

Middleware keeps retry logic out of the job body. Three are worth knowing. WithoutOverlapping takes a lock so two jobs for the same key never run together. RateLimited holds jobs back when a named limiter is exhausted, which is the correct way to respect a third-party API quota. ThrottlesExceptions backs off after repeated failures from a flaky service instead of burning all your tries in a minute. Both of the first two use the cache, so a shared cache store matters if you ever run more than one worker process on separate machines; the Valkey you already have is a good choice.

public function middleware(): array{    return [(new WithoutOverlapping($this->invoiceId))->releaseAfter(60)];}

For work that must never happen twice at the same time, ShouldBeUnique on the job class stops a second copy being queued while one is pending. It also relies on a cache lock.

Dispatching inside transactions#

This is the failure that shows up only under load. A controller opens a database transaction, creates an order, dispatches ProcessOrder, and commits a few milliseconds later. A worker on Valkey pops the job in under a millisecond, queries for order 812, and finds nothing, because the commit has not happened. The job throws ModelNotFoundException and retries; by the second attempt the row exists and it passes. You get a failed attempt in the logs and no explanation.

There are three fixes, in order of preference:

1. Set 'after_commit' => true on the redis connection in config/queue.php, or REDIS_QUEUE_AFTER_COMMIT=true if your config reads it. Laravel then holds the dispatch until the surrounding transaction commits and drops it if it rolls back.
2. Call ->afterCommit() on a single dispatch: ProcessOrder::dispatch($order)->afterCommit();
3. Implement ShouldQueueAfterCommit on the job class.

Prefer the first for a whole application, because forgetting the second on one dispatch is how the intermittent bug returns. The catch: with after_commit on, a dispatch outside any transaction behaves normally, so it costs nothing in the common case.

Failed jobs and the failed_jobs table#

When a job exhausts its tries, Laravel writes it to the failed-jobs store. That store is a database table, regardless of the queue driver, so a redis queue still needs a database for its failures. In a modern install the migration exists already; otherwise:

$ php artisan make:queue-failed-table$ php artisan migrate --force

The relevant block in config/queue.php chooses the driver, usually database-uuids, and the table name:

'failed' => [    'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),    'database' => env('DB_CONNECTION', 'mysql'),    'table' => 'failed_jobs',],

Each row holds the connection, the queue, the serialised payload, the exception with its stack trace, and the failure time. The commands you will use:

Before queue:retry all, read what failed. If forty jobs failed because a third-party API was down, retrying them is right. If forty failed because of a bug in your code, deploy the fix first, or you will watch all forty fail again and land back in the table with fresh timestamps.

Pruning is not automatic. A table that nobody trims grows with every incident, and it holds serialised payloads, which can include personal data. Schedule the cleanup so it happens without anyone remembering:

use Illuminate\Support\Facades\Schedule;Schedule::command('queue:prune-failed --hours=168')->daily();Schedule::command('queue:prune-batches --hours=48')->daily();

For per-job reactions, define a failed method. It runs once, after the final attempt, and is where you notify someone or undo a half-finished state:

public function failed(?Throwable $exception): void{    Log::error('Invoice send gave up', ['invoice' => $this->invoiceId, 'error' => $exception?->getMessage()]);}

For application-wide alerting, Queue::failing() in a service provider fires for every failure on every queue. Alert on the rate, not the individual event: one failure is a retry doing its job; thirty an hour is an incident. php artisan queue:monitor redis:default,redis:high --max=100 fires a QueueBusy event when a queue passes the threshold, and scheduling it every minute gives you a backlog alarm without a dashboard.

Run the worker: flags that matter#

The command from the first section is complete, but each flag has a reason. A reasonable production line:

$ php artisan queue:work redis --queue=high,default --tries=3 --backoff=10 \    --timeout=60 --max-time=3600 --max-jobs=1000 --memory=128 --sleep=3

The exits are deliberate. A PHP worker is a process that loads your application once and then runs jobs inside it, so any leak in a package, any static cache and any growth in the container's service bindings accumulates. --max-time, --max-jobs and --memory turn a slow leak into a clean exit every so often, and the supervisor restarts it with a fresh process. The corollary is critical: a worker that exits on purpose only works if something starts it again. That is the next section.

The same long-running behaviour is why a deploy must restart workers. php artisan queue:restart writes a timestamp into the cache; each worker checks it between jobs, finishes its current job and exits. It needs a working cache store, and a CACHE_STORE=array in production quietly makes the command do nothing.

Keep the worker alive in one container#

On a plain Linux server you would use systemd or Supervisor. In a single container you have one main process, and everything else has to live beside it or under it. A worker that is not supervised fails in a particular way: it exits after --max-time, an exception in a package, or a memory stop, the web process carries on serving pages, and jobs pile up in Valkey while the site looks healthy.

On RE:NODE each server is one container with a hard CPU limit and a memory limit. At the memory limit the container is stopped and restarts clean rather than swapping, so a worker that grows past what you left it kills the whole server, web process included. The platform also watches for crash loops: three restarts in an hour puts a warning on the server page and opens a ticket automatically, and six suspends the server. That makes the sizing below worth doing, and it makes a worker that exits and restarts itself by design a better pattern than one that crashes the container.

There are three honest options, from simplest to most capable.

Option 1: a second server for the worker

Because the queue lives in Valkey and not in the container, the worker does not have to share one with the web process. Run the same repository on a second Laravel / PHP server whose only job is queue:work, with the same .env values for the database and Valkey. Web traffic and job traffic then cannot starve each other, and each has its own memory and CPU limit. It also gives you the cleanest supervision story: the process the server runs is the worker, and when it exits the platform restarts the server. A Laravel / PHP app plan starts at $4 a month, and the Valkey server is reached over the network by both. The cost is a second .env to keep in step, and the start command of the second server needs to run the worker rather than the web server - check the Startup tab for what your plan lets you change.

Option 2: a loop script beside the web process

When one server has to do both, a small shell loop restarts the worker whenever it exits, and handles the stop signal so a restart does not cut a job in half.

#!/bin/sh# worker-loop.sh - restart queue:work whenever it exits, stop cleanly on SIGTERMstop=0trap 'stop=1; kill -TERM "$pid" 2>/dev/null' TERM INTwhile [ "$stop" -eq 0 ]; do  php artisan queue:work redis --queue=high,default --tries=3 --backoff=10 \    --max-time=3600 --memory=128 &  pid=$!  wait "$pid"  [ "$stop" -eq 0 ] && sleep 3donewait "$pid"

The first wait returns when the worker exits for any reason, and the sleep 3 stops a worker that fails at once from spinning the CPU. When a stop signal arrives, the trap forwards SIGTERM to the worker and the final wait lets it finish its current job. Laravel's worker handles SIGTERM by finishing the job it is on, then exiting. The script has to be started alongside the web process, which means your start command needs to launch both; where you cannot change it, this option is not available and Option 1 is the answer.

Option 3: Supervisor inside the container

If the container's start command can run supervisord in the foreground, one config file starts and restarts every process. This is the standard Laravel production setup, and it works the same in a container as on a server.

[supervisord]nodaemon=true[program:laravel-worker]process_name=%(program_name)s_%(process_num)02dcommand=php /app/artisan queue:work redis --queue=high,default --tries=3 --max-time=3600autostart=trueautorestart=truestopasgroup=truekillasgroup=truenumprocs=2redirect_stderr=truestdout_logfile=/app/storage/logs/worker.logstopwaitsecs=3600

autorestart=true restarts the worker on any exit, including the deliberate one from --max-time. stopwaitsecs must be longer than your longest job, or Supervisor kills a job in progress when it stops. numprocs=2 runs two workers; more is not better, as the next section shows. Supervisor itself has to be installed in the image or via your package manager, and it adds a process that you must also keep alive - the reason Options 1 and 2 are often easier on a small plan.

If your queue work is substantial, Laravel Horizon replaces the worker command with php artisan horizon. It supervises its own worker processes, balances them across queues, and adds a dashboard, so you run one process instead of several. It needs the redis driver, so it only applies once you are on Valkey. Run php artisan horizon:terminate in the deploy script, in the same place as queue:restart.

Graceful shutdown and health checks explains why the stop signal matters on every platform, and PM2 process manager guide covers the same supervision problem for Node.

Sizing workers in a small container#

A PHP worker holds a full copy of your framework. A bare Laravel worker is commonly in the region of 40 to 80 MB, a worker with a heavy package or a large collection in memory is several times that, and the real figure is whatever ps tells you on your own app. Measure instead of trusting a table:

$ ps -o pid,rss,cmd -C php | sort -k2 -n -r | head

rss is in kilobytes. Take the largest worker, multiply by the number of workers, add the web process (FrankenPHP or PHP-FPM workers each hold their own copy too), and compare the total with the plan's memory limit. A 1 GB plan comfortably holds a web process and two workers for ordinary jobs. Four workers that each peak at 150 MB is 600 MB before the web process has served a request.

Set --memory below what you can afford, not above. At the container's limit the whole server restarts, losing in-flight web requests and every running job. At the worker's own --memory limit only that worker exits and is restarted, and Laravel finishes the current job first.

CPU is the other limit. The container's CPU is a hard throttle to the share you bought, so a half-core plan running two CPU-heavy workers does not go faster with the second; it gets the same CPU split in two. A queue of image resizing wants workers equal to the cores you have. A queue of API calls and email is waiting on the network, not the CPU, and a single process can usefully hold several of them - at which point more worker processes is the way to get concurrency in PHP, since a worker runs one job at a time.

Putting the Valkey side on its own server also means its memory is not yours to spend. A Valkey plan is a separate server reached on its own host and port, from 256 MB, password-protected, and saved to disk with both AOF and snapshots, so pending jobs survive a restart of either side. The settings that decide whether a queue loses jobs, maxmemory-policy noeviction above all, are in Valkey memory and eviction policies, and the persistence modes are in Valkey persistence: AOF and RDB. Check yours with valkey-cli ... CONFIG GET maxmemory-policy and, if it is not noeviction, change it or ask whoever runs the server.

Deploys and the queue#

A deploy touches the queue in two ways: new code must reach the workers, and the jobs already in Valkey must still deserialise against the new code. The second is the one people forget.

A queued job stores its class name and its properties. If you rename a class or remove a property while jobs for it are waiting, the worker fails to unserialise them and they land in failed_jobs. The safe order is to deploy additive changes first: add the new class or property, let the queue drain of old jobs, then remove the old one in a later release. If you must rename, keep the old class as a thin subclass for a deploy or two.

On RE:NODE the deploy flow helps with the first problem. With GitHub deploy on push, a push restarts a running server (and starts a stopped one), so workers start again on the new code; see the deploy guide for the switches. The restart is not a pause, though: a clean stop lets a running job finish, a kill does not, and a job interrupted that way returns to the queue after retry_after seconds. That is the at-least-once behaviour again, and it is why jobs must be safe to repeat. Keep php artisan queue:restart in your deploy script anyway; it costs nothing and covers the hosts where a deploy does not restart the process. The rest of a deploy sequence is in Deploy Laravel to production.

Troubleshooting#

Jobs are pushed and nothing happens. No worker is running, or it is connected to a different queue or prefix. Run php artisan queue:work redis --once by hand and watch the output. Then compare QUEUE_CONNECTION and REDIS_PREFIX on the web process and the worker; a mismatch is invisible from the code.

`Connection refused` or `NOAUTH Authentication required`. The first is the host or port, or a firewall; the second is a missing REDIS_PASSWORD. Note that a cached config will ignore a corrected .env: run php artisan config:clear.

The job runs twice. retry_after is shorter than the job's runtime. Raise it above the job $timeout, as described earlier, or move the job to a queue with a longer limit.

`ModelNotFoundException` on some jobs only. The job was pushed before its transaction committed. Set after_commit to true on the connection.

The worker runs old code after a deploy. It was not restarted. Run php artisan queue:restart and make sure the cache store it relies on works.

Memory grows until the container restarts. A leak in a package or an unbounded collection. Add --max-jobs and --memory, and check ps for the real size. Stream large result sets with lazy() or cursor() instead of loading them.

`MaxAttemptsExceededException` and the job never ran its failure code. The worker was killed or timed out repeatedly and the attempts were used up. Check that $timeout is shorter than retry_after, and that the worker is not being killed by the container's memory limit.

`OOM command not allowed when used memory > 'maxmemory'`. Valkey is full and noeviction is refusing writes. Workers are down, too slow, or queues:*:reserved is full of stuck jobs. Fix the cause; do not change the eviction policy to make the error go away.

FAQ#

Do I need Horizon to run Laravel queues on Valkey?

No. php artisan queue:work is enough, and plain workers are easier to run in a small container. Horizon is worth adding when you want a dashboard, per-queue balancing, and metrics such as wait time and throughput without writing them yourself.

Can I keep the failed jobs table in Valkey too?

Not in a supported way. Failed jobs are stored through the failed block in config/queue.php, and its drivers are database, database-uuids, dynamodb and file. Keep them in your normal database, where you can query and prune them.

How many workers should I start in one container?

Start with one or two and measure. Each worker runs one job at a time, so for I/O-bound work a second worker doubles throughput, but memory and CPU are shared with the web process and do not grow with the count. Add a worker only when the oldest waiting job is older than you can accept.

Why does queue:retry all bring the same jobs back as failed?

Because the cause is still there. A retry runs the same payload against the current code and environment. Fix the bug or restore the dependency first, then retry; read the exception in queue:failed or the exception column before pushing anything back.

Is it safe to share one Valkey between the queue, cache and sessions?

It works, but it is the wrong shape. A cache wants to evict old keys and a queue must never lose one. Give the queue its own Valkey, or at least run with noeviction and accept that a full cache then refuses writes. Sessions in Valkey covers the same trade-off from the session side.

What happens to waiting jobs when the container restarts?

Pending jobs are in Valkey, not in the container, so they survive. Jobs a worker had reserved return to the queue after retry_after seconds and run again, which is why a job must be safe to repeat. Valkey saving to disk protects the rest if Valkey itself restarts.