Queues & Async Processing

Message queues let you process work asynchronously, absorb traffic spikes, and decouple services. They're the difference between a system that falls over under load and one that degrades gracefully.

Why queues exist

The core problems queues solve.

• ›Absorb traffic bursts — queue fills up instead of crashing the service
• ›Decouple producer and consumer — each scales independently
• ›Retry failed jobs — if a worker crashes, the message stays in the queue
• ›Async processing — don't make the user wait for slow operations
• ›Fan-out — one event triggers multiple downstream processes

Queue patterns

How queues are commonly used in system design.

• ›Task queue — workers pull jobs and process them (email sending, image resizing)
• ›Event streaming — append-only log, consumers track their own offset (Kafka)
• ›Pub/Sub — publish to a topic, multiple subscribers receive it (SNS, Pub/Sub)
• ›Dead letter queue — messages that fail repeatedly go here for inspection

Key tools

What to name in an interview.

• ›Redis (as queue) — simple, in-memory, fast. Not durable by default.
• ›RabbitMQ — traditional message broker, good for task queues
• ›AWS SQS — managed queue, at-least-once delivery, dead letter support
• ›Apache Kafka — distributed event log, replay-able, massive throughput
• ›Use Kafka when you need replay, audit trail, or high-volume event streaming

Interview tips

• ✓Add a queue whenever a write triggers slow downstream work
• ✓Specify delivery semantics: at-most-once, at-least-once, exactly-once
• ✓Address idempotency — what if a message is processed twice?
• ✓Kafka comes up in notification systems, feed generation, and analytics pipelines

Follow-up questions to expect

• ?How do you handle duplicate message processing?
• ?What happens if the queue fills up faster than consumers drain it?
• ?When would you choose Kafka over SQS?