Redis as a queue

Redis lists support atomic push and pop operations from both ends, which makes them a natural fit for job queues. There is no extra software to install, no broker to configure, and operations run in microseconds. The trade-off is that you have to manage reliability yourself.

The basic pattern

A FIFO queue uses two commands: LPUSH to add work to the left of the list, and RPOP to take work from the right.

127.0.0.1:6379> LPUSH jobs '{"type":"email","to":"alice@example.com"}'
(integer) 1
127.0.0.1:6379> LPUSH jobs '{"type":"email","to":"bob@example.com"}'
(integer) 2
127.0.0.1:6379> RPOP jobs
"{\"type\":\"email\",\"to\":\"alice@example.com\"}"

RPOP returns and removes the oldest item (the one pushed first). Both operations are O(1). The list key is created automatically on the first LPUSH and deleted automatically when the last item is popped.

Blocking dequeue with BRPOP

Polling with RPOP in a loop wastes CPU cycles when the queue is empty. BRPOP blocks the connection until an item is available, then returns it immediately.

127.0.0.1:6379> BRPOP jobs 30
1) "jobs"
2) "{\"type\":\"email\",\"to\":\"alice@example.com\"}"

The second argument is a timeout in seconds. If no item arrives within 30 seconds, BRPOP returns a nil reply. A timeout of 0 blocks indefinitely.

BRPOP accepts multiple keys. It returns the first item available, from whichever key has one:

127.0.0.1:6379> BRPOP high-priority jobs 30

Reliable queue pattern

BRPOP removes the item from the list before your worker processes it. If the worker crashes mid-job, the item is gone.

The reliable pattern moves the item to a separate "processing" list atomically instead of removing it outright. Use RPOPLPUSH (or LMOVE in Redis 6.2+) for this:

127.0.0.1:6379> RPOPLPUSH jobs jobs:processing
"{\"type\":\"email\",\"to\":\"alice@example.com\"}"

The item is now in jobs:processing. When the worker finishes successfully, it removes the item from jobs:processing. If the worker crashes, a recovery process can scan jobs:processing, check for stale items, and requeue them to jobs.

# Recovery: move stuck jobs back to the main queue
127.0.0.1:6379> RPOPLPUSH jobs:processing jobs

For blocking behavior, use BRPOPLPUSH (deprecated in Redis 6.2, still functional).

Using LMOVE (Redis 6.2+)

LMOVE is the modern replacement for RPOPLPUSH. It gives you explicit control over which end to pop from and which end to push to.

127.0.0.1:6379> LMOVE jobs jobs:processing LEFT RIGHT
"{\"type\":\"email\",\"to\":\"alice@example.com\"}"

LEFT RIGHT pops from the left of the source and pushes to the right of the destination. The blocking version is BLMOVE:

127.0.0.1:6379> BLMOVE jobs jobs:processing LEFT RIGHT 30

LMOVE is clearer about intent and works with the same performance characteristics as RPOPLPUSH. Prefer it on Redis 6.2+.

Python worker example

A realistic producer pushes JSON payloads that include a job type, a payload, and a creation timestamp. The consumer blocks on BRPOP in a loop.

Producer:

import json
import time
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def enqueue(job_type: str, payload: dict):
    job = {
        "type": job_type,
        "payload": payload,
        "created_at": time.time(),
    }
    r.lpush("jobs", json.dumps(job))

enqueue("send_email", {"to": "alice@example.com", "subject": "Welcome"})
enqueue("resize_image", {"url": "https://example.com/photo.jpg", "width": 800})

Consumer:

import json
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def process(job: dict):
    print(f"Processing {job['type']}: {job['payload']}")
    # Your job logic here

while True:
    result = r.brpop("jobs", timeout=30)
    if result is None:
        # Timeout: no jobs available, loop and block again
        continue
    _, raw = result
    job = json.loads(raw)
    process(job)

brpop returns a tuple of (key, value) when an item is available, or None on timeout. Destructuring with _, raw discards the key name.

Priority queues

Run multiple lists and have the worker check the high-priority queue first. BRPOP accepts multiple keys and returns from the first one that has an item:

127.0.0.1:6379> BRPOP jobs:high jobs:normal jobs:low 30

Redis checks the keys in order. If jobs:high has an item, it is returned first, regardless of what is in jobs:normal or jobs:low. In Python:

result = r.brpop(["jobs:high", "jobs:normal", "jobs:low"], timeout=30)

Producers push to the appropriate key based on job priority. The worker code stays the same.

Delayed jobs

For scheduled or delayed execution, use a sorted set. Store the job payload as the member and the Unix timestamp for when it should run as the score.

127.0.0.1:6379> ZADD jobs:scheduled 1748000000 '{"type":"send_report","payload":{}}'

A polling loop checks for jobs that are due:

import time
import json
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

while True:
    now = time.time()
    due = r.zrangebyscore("jobs:scheduled", "-inf", now, start=0, num=10)
    for raw in due:
        removed = r.zrem("jobs:scheduled", raw)
        if removed:
            # Only process if zrem succeeded (avoids double-processing with multiple workers)
            r.lpush("jobs", raw)
    time.sleep(1)

ZRANGEBYSCORE fetches jobs with a score (due time) at or before now. ZREM removes each one atomically. If two workers race on the same job, only the one that gets a non-zero return from ZREM wins.

When to use a dedicated queue instead

Redis lists are a good queue when:

• Your jobs are short-lived and tolerant of occasional loss
• You have Redis persistence enabled (RDB or AOF) and understand the recovery gap
• You want minimal infrastructure

Redis queues are a poor fit when:

• You need a dead letter queue. Failed jobs have to be handled manually unless you build the retry logic yourself.
• You need message acknowledgment. The reliable pattern approximates this, but it is not built in.
• You need routing or fanout. Lists deliver to one consumer. Routing to multiple queues based on job content requires external logic.
• You cannot tolerate any message loss. A Redis crash between a write and a flush can lose data even with AOF enabled.

Alternatives worth knowing:

Quick reference