Using Kafka as a message queue

Apache Kafka can act as a message queue, but it is not built like one. A traditional queue hands a message to one consumer and deletes it; Kafka appends messages to a partitioned log on disk, keeps them for a set retention period, and uses consumer groups so that only one member reads each partition. You get queue semantics and keep the ability to replay.

That difference is the whole story, and it cuts both ways. This guide covers what Kafka gives you over a conventional message broker, how to produce and consume from a topic as if it were a queue, and which queue features you have to build yourself. For a straight architectural comparison, see Kafka vs message queue.

Building webhooks?

Svix is the enterprise ready webhooks sending service. With Svix, you can build a secure, reliable, and scalable webhook platform in minutes. Looking to send webhooks? Give it a try!

Where traditional queues run out of room

Brokers like RabbitMQ and ActiveMQ are a good fit for most workloads, and at smaller scale they are easier to operate than Kafka. Three things push teams off them:

1. Horizontal scaling: adding capacity usually means a bigger broker or a cluster that shares queue state, rather than more independent consumers.
2. Durability: a message acknowledged but not yet persisted can be lost when a broker dies.
3. Retention: messages are deleted once delivered, so you cannot replay yesterday's traffic to debug a consumer bug or backfill a new service.

Kafka's log-based design addresses all three, at the cost of more moving parts. If you need the comparison in detail, we cover Kafka vs RabbitMQ separately.

What Kafka gives you that a queue does not

Kafka combines queue semantics with a distributed log, which buys you:

1. Partitions: a topic is split into partitions, and consumers read from them in parallel. Partition count is your unit of concurrency, so choosing it well matters; see our Kafka partition strategy guide.
2. Durability: messages are written to disk and replicated across brokers before they are acknowledged.
3. Retention: Kafka keeps messages for a configurable period even after they have been consumed, so a consumer can rewind its offset and reprocess.
4. Consumer groups: many consumers can read the same topic. Within a group each partition goes to exactly one member, which is where the queue behavior comes from.
5. Throughput: sequential disk writes and batching let a modest cluster handle hundreds of thousands of messages per second.

The cost is that Kafka has no per-message acknowledgement, which shapes most of the workarounds further down this page.

A worked example: order processing

Take an e-commerce application where placing an order has to trigger several unrelated jobs. The order service publishes one event to the orders topic and stops caring what happens next. An inventory service, a notification service, and a billing service each read that topic under their own group_id, so all three see every order and each one processes it once. Adding a fourth consumer later means deploying it with a new group, not changing the producer, and because Kafka keeps the log around, that new service can start from the beginning of the retention window instead of only seeing orders placed after it shipped.

Setting up Kafka as a message queue

Step 1: Install and start Kafka

Download Kafka from the Apache Kafka website. Recent releases run in KRaft mode and coordinate the cluster themselves, so a single-node setup is one command:

bin/kafka-server-start.sh config/kraft/server.properties

If you are on Kafka 3.2 or older, start Zookeeper first and then point the broker at it:

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Step 2: Create a topic

A topic is the closest thing Kafka has to a queue. Create one named orders:

bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2

• --partitions 3: Enables parallelism by dividing messages across three partitions.
• --replication-factor 2: Ensures fault tolerance by replicating data to two brokers.

Step 3: Produce messages

Write a producer to send messages to the orders topic. Use the kafka-python library to interact with Kafka.

Install the library:

pip install kafka-python

Create a producer script:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# Send order events
order_event = {'order_id': '12345', 'user_id': '67890', 'amount': 100.0}
producer.send('orders', value=order_event)

print(f"Order event sent: {order_event}")
producer.close()

Step 4: Consume messages

Consumers read from a topic and track their position with an offset. Create a consumer script:

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers=['localhost:9092'],
    value_deserializer=lambda v: json.loads(v.decode('utf-8')),
    auto_offset_reset='earliest',  # Start reading from the beginning
    group_id='order-processor'
)

print("Listening for order events...")
for message in consumer:
    print(f"Received order: {message.value}")

Step 5: Scale out with consumer groups

Run multiple instances of the consumer script with the same group_id and Kafka distributes partitions among them, one partition per member. That gives you the queue property you want: each message is handled once within the group.

The ceiling is the partition count. A topic with three partitions supports at most three useful consumers in a group, and any extra instances sit idle. Size partitions for the concurrency you expect to need, because raising the count later reshuffles how keys map to partitions.

Step 6: Watch what the cluster is doing

Use the Kafka CLI to inspect topics and consumer lag:

1. List topics:

   bin/kafka-topics.sh --list --bootstrap-server localhost:9092

2. Describe a topic:

   bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

3. Check consumer lag:

   bin/kafka-consumer-groups.sh --describe --group order-processor --bootstrap-server localhost:9092

The LAG column in that last command is the number you actually watch in production. Growing lag on one partition usually means an unbalanced partition key rather than an undersized cluster.

Filling the gaps a real queue would cover

Four things you have to add yourself when a Kafka topic stands in for a queue:

1. Offset commits: Kafka tracks consumption by offset, not per message. Turn off auto-commit and call consumer.commit() after your handler succeeds, or a crash mid-handler silently skips the message.
2. Exactly-once delivery: set enable_idempotence=True on the producer and use transactions when a consumer reads and writes in the same step.
3. Poison messages: because offsets advance in order, one message that always throws will stall its partition. Send it to a dead letter queue topic and move on; our Kafka DLQ guide has a working implementation.
4. Partition keys: pass a key to keep related messages ordered relative to each other.

   producer.send('orders', key=b'user_67890', value=order_event)

Retention is worth setting deliberately rather than leaving at the default seven days. A day is plenty for queue-style work and keeps disk usage predictable:

bin/kafka-configs.sh --alter --entity-type topics --entity-name orders --add-config retention.ms=86400000

Is Kafka the right queue for you?

Use Kafka as a queue when you want replay, when several independent services need the same event stream, or when throughput is high enough that a single-broker queue would become the bottleneck. Reach for RabbitMQ or SQS instead when you need per-message acknowledgement, priority ordering, or delayed delivery, since those are queue primitives Kafka has no answer for.

Kafka is also the wrong tool for pushing events to someone else's server. Consumers have to connect to your cluster and hold a client, which is not something you can ask your customers to do. That job belongs to webhooks, and the two pair well: consume from Kafka internally, then fan out to customer endpoints. Svix handles that outbound half, including retries, signature verification, and delivery logs. We compare the models directly in webhooks vs Kafka.