Message Queue Patterns and Use Cases

The Message Queues lesson showed how a broker delivers one message to one consumer. A consumer acknowledges (acks) each message after processing it. Because the broker redelivers unacknowledged messages, a consumer can receive the same message twice. Each pattern below starts from a failure in one system: a hotel booking service that calls its dependencies directly.

The Booking Service Without a Queue

Customers book hotels through a website. A load balancer distributes incoming requests across web servers, which call the booking service to write each booking to a relational database. After recording the booking, the booking service calls three other services. It asks a voucher service to generate a PDF voucher, an email service to send it, and a notification service to alert other systems.

Each "sync call" arrow represents a call that waits for an answer. The customer waits for all of them to finish, and a failure in any downstream service fails the entire booking. The sections below remove these synchronous waits one at a time.

Background Tasks

The booking service performs four tasks per booking. The durations below are illustrative:

TaskTime (ms)Must finish before the response?
Update the booking status in the database30Yes
Create a PDF voucher1,000No
Send a confirmation email with the voucher attached500No
Notify other services about the booking1,000No

Run sequentially, the request takes 30 + 1,000 + 500 + 1,000 = 2,530 ms. Only the database write must finish before the customer sees "Booking confirmed". The remaining three tasks can complete a few seconds later without affecting the customer.

A background task is work that the request handler starts without waiting for it to finish. The booking service writes the booking to the database (30 ms), puts one message per remaining task onto a queue, and returns the response. Enqueueing a message is a single short write to the broker, so the response time drops to about the 30 ms database write plus the broker write. Separate worker processes read the queued messages and do the slow work.

Because the confirmation email requires the PDF, those two tasks run in sequence. The worker that generates the PDF enqueues the email task when it finishes. The notification task does not depend on either step, so a second worker can run it in parallel.

Why Not a Background Thread?

A simpler design spawns a background thread inside the web process for each slow task and returns the response without waiting. The response is just as fast, but the pending work lives only in the web process's memory. Three problems follow:

  • A crash or a deploy loses the work. Deploying new code stops the old web processes. Any PDF that a thread was building is lost, and nothing records that the task ever started.
  • No retry. If the email provider goes down for a minute, the thread fails, and no component knows to resend the email later.
  • No visibility. Because tasks exist only as in-memory threads, no component can report how many tasks are waiting or failing.

A queue resolves all three issues by storing each task as a message in the broker. As explained in the Message Queues lesson, the broker retains each message until a worker acks it, so a crashed worker's task is redelivered. The queue length exposes the current backlog, and workers scale independently of the web servers.

Thread in the web processTask queue and workers
Web process crashes or redeploysTasks in progress are lostMessages stay in the broker
Task failsLost unless the code retriesRetried or moved aside (see below)
Backlog visibleNoQueue length
Scale the slow workAdd web serversAdd workers

Work Queues and Competing Consumers

One worker builds about one PDF per second. Suppose a busy hour brings an illustrative rate of 5 bookings per second. The PDF queue then grows by 5 − 1 = 4 messages every second, delaying each customer's voucher further.

A work queue distributes messages from a single queue across several identical workers. These workers act as competing consumers: the broker delivers each message to only one worker, so adding workers increases throughput without duplicating work. With 5 workers, the queue matches the incoming rate of 5 bookings per second. With 6, it also drains an existing backlog at about 1 message per second.

Competing consumers introduce two constraints. First, messages finish out of order, because worker 2 can finish task 2 before worker 1 finishes task 1. Work queues therefore suit independent tasks, such as generating vouchers for separate bookings. Second, because any worker can receive any message, workers must not hold local state that a later message requires.

Publish/Subscribe Fan-Out

While a work queue delivers each message to a single consumer, a booking event must reach several services. The email service, push notification service, and analytics service each require their own copy of every "booking confirmed" event.

Publish/subscribe (pub/sub) delivers a single published message to every subscriber. The producer publishes the event to a topic (a named channel for one kind of event). The broker copies it into a dedicated queue for each subscribing service, and each service reads only from its own queue.

Giving each subscriber its own queue keeps the services isolated. If the analytics service goes down for an hour, its queue accumulates messages while email and push notifications continue at their normal pace. When analytics recovers, it drains the backlog from its queue. Each subscribing service can also scale its own worker pool as a work queue.

The producer does not know which services subscribe to the topic. To award loyalty points for each booking, the loyalty team creates a fourth queue and subscribes it to the topic without modifying the booking service. In the synchronous design from the first diagram, adding a subscriber requires a new call in the booking service and introduces a new point of failure.

Different brokers use different names for these components. In RabbitMQ, the producer publishes to a fanout exchange, which copies each message to every bound queue. In AWS, an SNS topic fans out to SQS queues subscribed to that topic.

Work queuePub/sub with a queue per subscriber
Who receives a messageOne workerEvery subscribing service
PurposeShare the load of one taskTell several services about one event
Add a consumerMore capacity for the same taskA new service gets every event

Load Leveling During a Promotion

The booking service and its database have fixed write capacity, which a promotional traffic spike can exceed. In the direct design, web servers forward every request directly to the booking service, overloading the database until requests time out and customers see errors.

Load leveling places a queue between the web servers and the booking writers. Web servers enqueue each booking request and return "Booking received", while booking workers pull from the queue at a rate the database can sustain. The queue absorbs the excess traffic during the spike and drains after traffic subsides.

The numbers in the diagram are illustrative. Suppose a promotion sends 2,000 booking requests per second for 5 minutes while the database supports 500 writes per second. The queue grows by 1,500 messages per second, reaching 450,000 messages after 300 seconds. Suppose traffic then drops to 100 requests per second. Workers drain the backlog at 500 − 100 = 400 messages per second, clearing it in about 1,125 seconds, or about 19 minutes.

This arithmetic highlights the trade-off: load leveling trades errors for processing delay. The customer sees "Booking received" right away, but final confirmation can take minutes, so the product must support a pending state. If incoming traffic exceeds drain capacity for hours, the queue only postpones the failure. The Master Template lesson includes a widget where a traffic burst fills a queue that later drains.

When Processing Fails

Every pattern above assumes that workers eventually succeed. Some failures are transient and resolve on their own. Others repeat on every attempt, requiring a separate handling strategy.

Retry With Backoff and Jitter

Suppose the email provider returns errors for 30 seconds during a deploy. A worker that retries without waiting fails repeatedly during that window, adding load to an already struggling service. Exponential backoff increases the wait between retries. For example, the waits can be 1 second, then 2, 4, and 8. Jitter adds a random offset to each wait so that a thousand workers failing together do not retry in lockstep. The Retries lesson covers both techniques in detail.

In a queue-based system, the consumer does not ack the failed message. Some brokers return it at once, as a RabbitMQ requeue does. Others make it visible again after a fixed delay, such as an SQS visibility timeout (see Message Queues). Backoff needs a delayed-retry feature in the broker or client library, or a consumer that sets the delay.

Poison Messages

One booking message lacks a hotel ID because of a bug in an older client, causing the voucher worker to throw an exception on every attempt. A message that fails on every delivery is a poison message. Retrying cannot fix it because the defect lies in the message payload or consumer code rather than a temporary condition.

Without a retry limit, a poison message cycles back to the queue forever, using worker time that valid messages need. The impact is worse when the broker returns failed messages to the head of the queue. RabbitMQ, for example, places a requeued message at its original position when it can, and its documentation warns that such requeue loops waste network bandwidth and CPU. With one worker, the poison message at the front blocks every message behind it.

Attempt Limits and the Dead-Letter Queue

Resolving poison messages requires two mechanisms. First, a maximum attempt count caps the retries: the broker or the worker counts the deliveries of each message. When a message reaches the limit, it moves to a dead-letter queue (DLQ), a separate queue for messages that could not be processed. The main queue then continues with the good messages. After the bug is fixed, redrive moves the dead-lettered messages back to the main queue.

The widget below runs this flow on a queue of bookings. booking-102 is the poison message, and each attempt takes one worker one second. Press Play with the default settings (one worker, no limit, immediate retry), and watch the queue grow behind booking-102. Then press Reset, set Max attempts to 3, and press Play again. Next, set Max attempts back to No limit and try 2 workers. Then try 1 worker with backoff, and compare the red share of worker time. When booking-102 is in the DLQ, press "Fix the consumer and redrive".

t = 0s
WorkersMax attemptsRetry
Booking service
adds 1 booking every 2 s
next: booking-107
→
Queue: 6 waiting
booking-101booking-102booking-103booking-104booking-105booking-106
head of the queue on the left
→
Workers
worker 1waiting
worker 2not running
0 bookings processed
Waiting to retry
not used: retries go straight back to the queue
Dead-letter queue: 0
not used: there is no attempt limit
Six bookings wait in the queue. booking-102 has no hotel ID, so it fails on every attempt. Press Play or "Step 1 second".
Worker time per second (last second):■ good bookings 0 s■ poison message 0 squeue depth 6poison attempts 0
One bar per second: 1 or 2 worker-seconds
One attempt takes one worker one second, and a booking arrives every 2 s. These rates and the limit of 3 attempts are illustrative.

With one worker and no attempt limit, the worker spends every second retrying booking-102 while the queue grows by one booking every 2 seconds. With a second worker, the other bookings are processed, but half of the worker time still goes to booking-102. Enabling backoff widens the gaps between retries and reduces wasted work, yet the poison message still returns forever. Only an attempt limit clears the blockage: after the third failure, booking-102 moves to the DLQ and the main queue drains.

Brokers build this behavior in directly. In Amazon SQS, a redrive policy configures maxReceiveCount, the number of receives before SQS moves a message to the DLQ. RabbitMQ quorum queues enforce a delivery limit that defaults to 20 since RabbitMQ 4.0. A message over the limit goes to a dead-letter exchange (RabbitMQ's route to a dead-letter queue) if configured, or is dropped otherwise.

A DLQ only helps if engineers monitor it. A typical setup triggers an alarm when the DLQ is not empty. An engineer reads the failed messages and the consumer's error logs, and fixes the underlying bug. Redrive then moves the messages from the DLQ back to the main queue for the fixed consumer to process. Because redriven messages are delivered more than once, consumers must handle duplicate deliveries safely. The Message Queues lesson shows how a consumer makes duplicates harmless, and the Idempotency lesson covers the general technique.

Setting the attempt limit requires balancing two risks. A limit set too low routes valid messages to the DLQ during brief dependency outages that a few backoff retries would survive. Moving a failed message aside also alters delivery order. If later messages depend on earlier ones, such as sequential edits to the same booking, the SQS documentation advises against a DLQ on a FIFO (first in, first out) queue where strict ordering matters.

Key idea. A work queue gives each message to one worker, and pub/sub gives each subscriber its own queue and a copy of every event. Load leveling turns a spike into a backlog that drains at the database's write rate. Retries with backoff handle transient failures, and an attempt limit moves a poison message to a DLQ for a fix and redrive.

The next lesson, the Redis Queue Tutorial, runs a background job on a real queue and shows the queue, job record, and failed-job registry in Redis.

Further Reading