Redis vs Kafka: Streams, Queues, and When to Use Each

Redis vs Kafka comes up when a team needs to move events between services and already runs one of the two. They overlap less than the question suggests. Redis is an in-memory data store that happens to have a good stream type. Kafka is a distributed, disk-based commit log built for nothing but streams.

The short verdict: use Redis Streams when event volume fits in memory, retention is measured in hours or a bounded number of entries, and you want one less system to run. Use Kafka when events are a durable record other teams will replay, when retention is days or longer, or when throughput needs to spread across many machines.

Plenty of teams run both: Kafka as the event backbone, Redis next to the services for caching, rate limits, and short-lived work queues.

Redis vs Kafka: the architectural difference

Everything else follows from where the data sits.

Redis keeps the dataset in RAM. A stream is a key like any other, stored in a radix tree, and every entry costs memory until you delete or trim it. Persistence is a separate choice: RDB snapshots, an append-only file, or both. The official redis:8 Docker image ships with AOF off and snapshotting on a schedule:

> CONFIG GET appendonly
1) "appendonly"
2) "no"
> CONFIG GET save
1) "save"
2) "3600 1 300 100 60 10000"

With those defaults a crash can lose the writes since the last snapshot. Turning on AOF with appendfsync everysec narrows that to about a second, which is fine for many event workloads and not fine for a payment ledger.

Kafka writes every record to an append-only log segment on disk and replicates it to other brokers before the producer gets an acknowledgement (producers default to acks=all since Kafka 3.0). Reads are sequential disk reads served mostly from the OS page cache. Because storage is disk, keeping a week or a year of events is a matter of buying disks, not RAM.

Since Kafka 4.0, the cluster runs in KRaft mode only. ZooKeeper is gone, so a modern Kafka deployment is brokers plus a small quorum of controllers, which can run in the same processes for small clusters.

Consumer groups: same name, different mechanics

Both systems have consumer groups, and they solve the same problem: several workers share a stream without processing the same event twice. The bookkeeping is where they differ.

Redis Streams consumer groups

Redis tracks delivery per entry. Every entry handed to a consumer goes into the group's pending entries list (PEL) until the consumer acknowledges it with XACK. Here are two workers in a billing group, run against Redis 8:

> XADD orders * customer customer-1 total 10
"1790758868234-0"
> XADD orders * customer customer-2 total 20
"1790758868290-0"
> XGROUP CREATE orders billing 0
OK
> XREADGROUP GROUP billing worker-1 COUNT 1 STREAMS orders >
1) 1) "orders"
   2) 1) 1) "1790758868234-0"
         2) 1) "customer"
            2) "customer-1"
            3) "total"
            4) "10"
> XREADGROUP GROUP billing worker-2 COUNT 1 STREAMS orders >
1) 1) "orders"
   2) 1) 1) "1790758868290-0"
         2) 1) "customer"
            2) "customer-2"
            3) "total"
            4) "20"
> XPENDING orders billing
1) (integer) 2
2) "1790758868234-0"
3) "1790758868290-0"
4) 1) 1) "worker-1"
      2) "1"
   2) 1) "worker-2"
      2) "1"

Each worker got a different entry, and Redis knows which worker holds which one. If worker-2 dies, another worker can take over its idle entries with XAUTOCLAIM. When I let two entries sit for a second and claimed them for worker-1, XPENDING showed the delivery count had gone up to 2, which is the number you use to push poison messages to a dead-letter stream.

A second group reads the same stream independently:

> XGROUP CREATE orders analytics 0
OK
> XREADGROUP GROUP analytics dashboard COUNT 10 STREAMS orders >
1) 1) "orders"
   2) 1) 1) "1790758868234-0"
         2) 1) "customer"
            2) "customer-1"
            3) "total"
            4) "10"
      2) 1) "1790758868290-0"
         2) 1) "customer"
            2) "customer-2"
            3) "total"
            4) "20"

Acknowledging an entry does not delete it. The stream keeps every entry until you trim it, which is why production streams almost always use XADD ... MAXLEN ~ n or a periodic XTRIM. Our Redis Streams guide covers trimming, claiming, and the work-queue pattern in more detail.

Kafka consumer groups

Kafka tracks progress per partition, not per record. Each partition is assigned to exactly one consumer in a group, and the group commits an offset: "everything before this position is done". I created a three-partition orders topic on Kafka 4.3, produced six keyed records, and consumed them with a billing group:

Partition:0	Offset:0	customer-1	{"order_id":1,"total":10}
Partition:0	Offset:1	customer-5	{"order_id":5,"total":50}
Partition:1	Offset:0	customer-3	{"order_id":3,"total":30}
Partition:1	Offset:1	customer-4	{"order_id":4,"total":40}
Partition:1	Offset:2	customer-6	{"order_id":6,"total":60}
Partition:2	Offset:0	customer-2	{"order_id":2,"total":20}
Processed a total of 6 messages

The record key decides the partition, so all events for customer-5 land in partition 0 and stay in order relative to each other. There is no global order across partitions. The group's state is three numbers:

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
billing         orders          0          2               2               0
billing         orders          1          3               3               0
billing         orders          2          1               1               0

This model is cheap. Kafka does not remember anything per record, so a group can consume millions of records per second and its state stays tiny. The cost is that parallelism is capped by the partition count: a topic with three partitions can keep at most three consumers in one group busy. It also means one slow record blocks the records behind it in the same partition.

Kafka 4.2 made share groups (KIP-932, "Queues for Kafka") production-ready. A share group lets several consumers read the same partition and acknowledge records one at a time, with delivery counts, which is closer to how Redis Streams and RabbitMQ behave. If you picked Redis over Kafka because you needed queue semantics, that reason is weaker than it was.

Replay and retention

Replay is where Kafka pulls ahead.

Kafka's default retention is seven days. The topic created above reports it directly:

retention.ms=604800000 sensitive=false synonyms={}

Replaying is an offset change. Resetting the billing group to the start of the topic puts all six records back in front of it:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group billing --topic orders --reset-offsets --to-earliest --execute

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
billing         orders          0          0               2               2
billing         orders          1          0               3               3
billing         orders          2          0               1               1

This is what makes Kafka a system of record. A new service can read months of history to build its own state. A bug fix can reprocess yesterday. Log compaction (cleanup.policy=compact) keeps the latest record per key forever, which turns a topic into a changelog, and tiered storage moves old segments to object storage so long retention does not mean large broker disks.

Redis can replay too: XRANGE reads any ID range, and XGROUP SETID moves a group back. The limit is that anything you keep costs RAM. A stream of 1 KB events at 5,000 events per second is about 430 GB per day before overhead. Nobody keeps that in Redis. In practice, Redis Streams retain minutes to days of data, bounded by MAXLEN or MINID.

DB Pro

Work With Your Databases Like A Pro

Query, explore, and manage your databases with a beautiful desktop app and built-in AI.

Download Now

Throughput and scaling

A single Redis stream lives on one key, and in Redis Cluster one key lives on one shard. You scale a Redis event pipeline by splitting events across several stream keys (for example orders:{0} to orders:{15}) and teaching producers and consumers about the split. That works, but it is the application's job.

Kafka does that split for you. Partitions spread across brokers, producers pick a partition by key, and adding brokers adds capacity once partitions are reassigned. Replication is per partition, and the cluster handles leader election when a broker fails.

Per event, Redis has less work to do: an XADD is an in-memory append, with no disk write or replica round trip before the reply unless you ask for one. Kafka's advantage is not latency. Its ceiling is far higher, and the data survives for as long as you are willing to store it.

Delivery guarantees

Both systems give you at-least-once delivery with consumer groups. Neither gives you exactly-once processing for free.

• Redis Streams: an entry stays in the PEL until XACK. If the consumer crashes, another consumer claims it. The durability of the entry itself depends on your persistence and replication settings. Replication to replicas is asynchronous; WAIT can block until a write reaches a number of replicas, but it does not make Redis a consensus system.
• Kafka: a record is committed once the in-sync replicas have it. Consumers commit offsets after processing. Idempotent producers are on by default, and Kafka transactions give exactly-once semantics for read-process-write pipelines that stay inside Kafka.

If you need fire-and-forget fan-out, Redis also has plain pub/sub, which is at-most-once: a message published while no one is subscribed goes nowhere. That is a different tool from Streams and not a Kafka competitor.

Operations

Redis is one process with a config file. A replica and Sentinel, or a managed service such as AWS ElastiCache, covers most availability needs. If you already run Redis for caching, adding a stream costs nothing operationally.

Kafka is a cluster. Even with KRaft removing ZooKeeper, you are running brokers, controllers, partition reassignments, retention and disk monitoring, consumer lag alerts, and client version upgrades. Managed Kafka (Confluent Cloud, Amazon MSK, Aiven, Redpanda's Kafka-compatible service, and others) removes most of that, and prices by some mix of throughput, storage, partitions, or cluster size.

The ecosystem gap is large. Kafka Connect has connectors for most databases and SaaS tools, Debezium streams database changes into Kafka, and stream processors (Kafka Streams, Flink, Spark) read Kafka natively. Redis has client libraries in every language and not much of a connector ecosystem around Streams.

Redis Streams vs Kafka: when to choose Redis

• You already run Redis, and the events are internal to one product.
• Retention is short: you process events and move on, or you need the last N entries for a feed.
• Volume fits in memory with room to spare, including a burst when consumers fall behind.
• You want per-message acknowledgement and claiming of stuck work without extra machinery.
• You want the lowest possible latency from produce to consume.

Typical fits are background jobs, notification fan-out, activity feeds, rate-limit events, and real-time dashboards. If the job is a work queue rather than an event log, also consider plain Redis lists, or a dedicated broker; RabbitMQ vs Redis covers that trade-off.

When to choose Kafka

• Events are a durable record: orders, payments, audit logs, user activity that analytics will read later.
• Several teams consume the same data at their own pace, and some of them will need to replay history.
• You are doing change data capture from a database, or feeding a warehouse or lake.
• Throughput or retention is beyond what one Redis shard can hold in memory.
• You want an ecosystem of connectors and stream processors rather than writing that code yourself.

Using Redis and Kafka together

The two combine well because they do different jobs. A common layout:

1. Services publish domain events to Kafka.
2. Consumers build read models and write hot data into Redis: session state, counters, leaderboards, cached API responses.
3. Short-lived coordination (locks, rate limits, a job queue for one service) stays in Redis and never touches Kafka.

If Redis is the part of the stack you inspect most, DB Pro's Redis desktop client shows streams, consumer groups, and keys in a GUI. For choosing between Redis-compatible servers, see Redis alternatives, and for the pure caching question, Redis vs Memcached.

Verdict

Redis vs Kafka is a question of how long your events live and how many machines they need.

Pick Redis Streams for fast, short-lived events inside one system, especially when Redis is already in the stack. Pick Kafka when events are a long-lived, replayable record that several consumers depend on, or when volume outgrows memory. If you are unsure, start by estimating a week of events in bytes. If that number fits comfortably in RAM and nobody needs last month's data, Redis will do the job with far less to operate.