Message queues: why and how

The problem with calling directly

After an order is placed, the orders service needs to send an email, credit bonuses, and update analytics. If it calls these three services directly and waits for a reply, it becomes slow and fragile: the email service goes down, and the order can't be placed, even though it has nothing to do with email. A message queue decouples this.

Producer, queue, consumer

The idea is simple. A service doesn't call others directly, it puts a message into a queue and moves on. Other services pick messages up from the queue and process them at their own pace.

• Producer puts messages in.
• Consumer picks them up and processes them.
• Broker stores messages between the two. The queue lives inside the broker.

The orders service no longer waits on email and no longer crashes because of it: it dropped off a message and is free.

What this buys you

• Decoupling. Producer and consumer don't know about each other and don't crash together. The consumer goes down, messages pile up in the queue and get processed once it's back.
• Smoothing spikes. A burst of orders arrives, the queue absorbs them, and the consumer works through them at its own pace. The database doesn't choke.
• Scaling. The queue is being worked through too slowly, you add more consumers on the same queue, and they process messages in parallel.

There's a cost too. The broker becomes one more service you need to keep alive and monitor. And the result of processing doesn't arrive right away: the email goes out a second later or a minute later, depending on the queue.

Acknowledgment: a message won't get lost

A consumer picks up a message and crashes mid-processing. If the broker already deleted the message, it's lost. That's why brokers wait for an acknowledgment (ack): the consumer reports "processed", and only then is the message considered delivered.

Until the acknowledgment arrives, the message sits as unacknowledged. If the consumer died, the broker hands this message out again, either to the same consumer after restart or to another one. In RabbitMQ, unacknowledged messages go back to the queue when the consumer's channel closes.

Hence the rule: acknowledge a message after the work is done, not before. Acknowledge before processing and then crash, and the message is lost.

At-least-once and idempotency

Acknowledgment protects against loss, but it creates repeats. A consumer credits a bonus, crashes before acknowledging, and after restart gets the same message a second time. This guarantee is called at-least-once: a message gets through at least once, but it may arrive again.

So a consumer needs to be idempotent, like the API from the idempotency lesson: processing the same message twice shouldn't credit the bonus twice. The RabbitMQ docs explicitly require consumers to be ready for redelivery.

A common approach: the message carries its own ID (an order number, an event ID), and the consumer remembers the IDs it has handled. An ID it has already seen means the consumer skips the work and just acknowledges the message. In Redis, remembering an ID is convenient with SET ... NX: it writes the key only if it doesn't exist yet.

SET done:event:77 1 NX    -- OK: first time, do the work
SET done:event:77 1 NX    -- (nil): already processed, skip

Best of all is when the "processed" marker and the work itself change together, in one transaction or one script. Otherwise there's a narrow window left: the marker is set, a crash happens before the work, and the message is skipped for good.

Queues by hand: Redis Streams

We'll handle queues in the labs with Redis, which you already know from the previous lesson. It has a data type called stream: a log of messages that get appended to the end. Each message gets an ID like 1791816841957-0 (a timestamp in milliseconds and a sequence number) and a set of fields.

XADD events * type signup user 7     -- append a message, * means "pick the ID yourself"
XLEN events                          -- how many messages in the stream
XRANGE events - +                    -- show them all, from first to last

Readers are organized into consumer groups. A group remembers how far it has read, and hands messages out to its consumers: each message within a group goes to exactly one consumer.

XGROUP CREATE events mailer 0                            -- group mailer, read from the very start
XREADGROUP GROUP mailer mailer-1 COUNT 10 STREAMS events >
XACK events mailer 1791816841957-0                        -- acknowledge by ID

XREADGROUP has two special values at the end. > asks for new messages the group hasn't handed out to anyone yet. 0 asks for its own unacknowledged ones: that's how a consumer, after a restart, picks up what it didn't manage to acknowledge.

A group's unacknowledged messages show up through XPENDING, and group state through XINFO GROUPS:

XPENDING events mailer           -- how many are pending and who has them
XINFO GROUPS events              -- per group: pending, last-delivered-id, entries-read, lag

Here pending is the number of messages handed out but not acknowledged. entries-read shows how many messages the group has read, and lag how many are still waiting for it in the stream. If a consumer dies for good, another one picks up its unacknowledged messages with XAUTOCLAIM.

After reading, a message doesn't disappear from the stream. Two groups on the same stream read it independently, and each gets every message. This is closer to Kafka than to a classic queue, which is exactly why a stream is a convenient way to show both approaches.

RabbitMQ and Kafka

The two most common brokers solve slightly different problems:

• RabbitMQ is a classic task queue. One consumer picks up a message, acknowledges it, and it leaves the queue. Good for "do the work once" (send an email, process a payment).
• Kafka is an event log. Messages aren't deleted after reading, they sit in the log for as long as the topic's retention setting says. Different groups read the log independently, each with its own position (offset). Good for event streams and analytics, where one event is needed by many.

If you need a task queue, you usually reach for RabbitMQ. If you need an event stream for many consumers, you reach for Kafka. The line isn't hard: RabbitMQ has a stream queue type with log-like retention, and Redis Streams can do both at a smaller scale.

A mapping of concepts, so you can carry experience over from the labs:

Redis Streams	RabbitMQ	Kafka
stream	queue	topic
consumer group	several consumers on one queue	consumer group
XACK	basic.ack	commit offset

Common mistakes

• Acknowledging before processing. The consumer acks the moment it gets the message, then crashes. The message is lost, and there won't be a redelivery.
• A non-idempotent consumer. A repeat after a crash credits the bonus a second time or sends two emails. Under at-least-once, a repeat isn't an incident, it's normal.
• Fixing the data instead of the code. You subtracted the extra bonus by hand, but left the consumer unchanged. The next crash produces the same bug.
• Forgetting your own unacknowledged messages. After restart, the consumer only reads >, and its old messages from XPENDING never get picked up by anyone. They hang forever.
• One group for everyone. Email and analytics read the stream in one group and split the messages between them: email gets one event, analytics gets another. Every independent job needs its own group.
• A message that always fails. The consumer crashes on one message, restarts, gets it again, and crashes again. After a few tries, such messages are moved to a separate queue (a dead letter queue) and sorted out by hand.

What you'll need for the tasks

• A Redis client inside the container: docker exec -it <container> redis-cli.
• The stream: XADD, XLEN, XRANGE <stream> - +.
• Groups: XGROUP CREATE <stream> <group> 0, XREADGROUP GROUP <group> <consumer> COUNT <N> STREAMS <stream> >, XACK <stream> <group> <ID>... (you can list several IDs separated by spaces).
• State: XINFO GROUPS <stream>, XPENDING <stream> <group>.
• An "already processed" marker: SET <key> 1 NX replies OK the first time and an empty reply after that.
• For the mini-project, the whole module: psql from the Postgres lesson, the training migrate.sh from the migrations lesson, the connection budget from the pools lesson, SET ... EX from the Redis lesson.

This wraps up the data track: database, migrations, pools, cache, and queues.