1. Courses
  2. System Design Fundamentals
  3. Lecture 5
Lecture 5 of 1419 minInteractiveFree lecture

Queues and asynchronous processing

Not every piece of work needs to happen while the user waits. Put a queue between producers and consumers, watch it absorb spikes and grow when consumers can’t keep up, crash a consumer, and see why idempotency matters.

In this lecture you will

  • Explain when to move work out of the request path into a queue
  • Relate arrival rate, consumer capacity and queue depth
  • Explain at-least-once delivery and why messages can be processed twice
  • Design idempotent consumers and handle poison messages with dead-letter queues

When a user places an order, several things must happen: charge the card, reserve stock, send a confirmation email, update analytics, notify the warehouse. If the web request waits for all of them, it is slow — and fails whenever any one of those systems is slow or down. Much of that work doesn’t need to finish before the user sees “Order confirmed”. It needs to happen reliably, soon. That is what queues are for.

Producers, queues and consumers

A producer — the web app — puts a message such as “order 8812 placed” on a queue and moves on. One or more consumers take messages off the queue and do the work. The queue sits in between and holds messages safely until someone processes them.

This decoupling brings several benefits:

  • Faster responses — the request does only what must happen now.
  • Resilience — if the email service is down, messages wait instead of failing orders.
  • Spike absorption — a burst of orders queues up and is processed at a steady pace.
  • Independent scaling — add consumers when the backlog grows, without touching the web tier.

Common tools include Amazon SQS, RabbitMQ and Google Pub/Sub for work queues, and Apache Kafka for high-volume event streams.

See the difference on a single order. Compare doing everything inside the request with queueing the slow work — then take the email service down:

InteractiveInside the request, or through a queueSwitch modes · take the email service down
Loading visual…

Durations are illustrative. The email step times out after 5 seconds, a common client default.

Watch the queue

Adjust the arrival rate and the number of consumers. Then crash a consumer while it is working, first with idempotent consumers and then without.

InteractiveA work queue with a pool of consumersChange rate and consumers, then crash one
Loading visual…

Each consumer handles about 1.1 messages per second. A crash after the work is partly done — say, the email already sent — returns the message to the queue.

The first lesson is simple arithmetic: whenever messages arrive faster than consumers can process them, the queue grows — steadily and without limit. Queues absorb temporary spikes, but the long-run consumer capacity must exceed the average arrival rate. Queue depth, and the age of the oldest message, are the metrics to alert on.

At-least-once delivery

A consumer receives a message, does the work and then acknowledges it, which deletes it from the queue. If the consumer crashes before acknowledging, the queue can’t know whether the work was done, so after a visibility timeout it delivers the message again. Most queues therefore guarantee at-least-once delivery: nothing is lost, but some messages are processed twice.

Idempotent consumers

The fix belongs in the consumer: make processing idempotent, so doing it twice has the same effect as doing it once.

def handle(message):
    with db.transaction():
        if db.exists("processed_messages", id=message.id):
            return                                   # already done — ignore the duplicate
        charge_card(message.order_id, message.amount, idempotency_key=message.id)
        db.insert("processed_messages", id=message.id)
    message.ack()

Record each message ID in the same transaction as the effect, and pass idempotency keys to external APIs — payment providers support them precisely for this reason.

Publishing reliably: the outbox

Duplicates are the consumer’s problem; missing messages start at the producer. Saving an order and publishing its event are two separate writes to two separate systems — and a crash between them leaves them disagreeing. The transactional outbox fixes this by making the event part of the database transaction:

AnimationThe dual-write problem and the transactional outbox
Loading animation…
  1. Two writesPlacing an order means saving it in the database and publishing an OrderPlaced event, which the email and warehouse services consume.
  2. A crash in betweenThe order commits, then the service crashes before publishing. The customer is charged, but no email is sent and nothing ships — and no error appears anywhere. Publishing first has the opposite bug: an email for an order that was never saved.
  3. The outboxInstead, write the event as a row in an outbox table, in the same transaction as the order. The database guarantees both rows are saved, or neither.
  4. The relayA separate relay process reads unsent outbox rows, publishes them to the queue and marks them sent. If it crashes, it resumes from the unsent rows when it restarts.
  5. At least onceIf the relay crashes after publishing but before marking the row sent, the event is published twice. That’s acceptable — because consumers are idempotent.

Change-data-capture tools such as Debezium can act as the relay by reading the database’s log.

Visibility timeout
How long a received message stays hidden before it is delivered again. Set it longer than normal processing time.
Dead-letter queue
Where messages go after failing several times, so one “poison” message doesn’t block the queue or retry forever.
Backpressure
Slowing producers, or shedding low-priority work, when consumers can’t keep up.
Ordering
Many queues don’t guarantee order. When order matters, use FIFO queues or partition by key, as Kafka does.

That completes the first module: requirements and estimates, load balancing, caching, replication and sharding, and queues — the building blocks of almost every large system. Module 2 covers consistency and CAP, API design, rate limiting, observability and service architecture; module 3 applies everything to real case studies.

Key takeaways

  • Queues decouple producers from consumers, smooth out traffic spikes and let each side scale independently.
  • If messages arrive faster than consumers process them, the queue grows without limit — capacity must exceed the average rate.
  • Most queues deliver at least once, so a crashed consumer’s message is delivered again.
  • Make consumers idempotent — processing a message twice must have the same effect as processing it once.

Check your understanding

3 questions · answers explained

Q1Messages arrive at 5 per second. Each consumer handles about 1.1 per second. What happens with 3 consumers?

Capacity is 3 × 1.1 ≈ 3.3 messages per second, less than 5 arriving. The backlog grows by about 1.7 messages every second until you add consumers or reduce the work per message.

Q2A consumer crashes after sending a customer email but before acknowledging the message. What happens next with at-least-once delivery?

Unacknowledged messages are redelivered after a timeout. Without idempotency, the side effect — the email — happens twice.

Q3What makes a payment consumer idempotent?

A unique ID per message (or per business operation), checked and recorded in the same transaction as the effect, turns duplicates into no-ops.