Message Queues

How it works

A message queue lets services communicate asynchronously: producers publish messages, consumers process them at their own pace. It absorbs traffic spikes, decouples components, and adds durability and retries — the backbone of event-driven architecture.

Mental models

• Asynchronous decoupling: the producer doesn't wait for the consumer.
• Queues buffer bursts so a spike doesn't topple downstream services.
• At-least-once delivery + idempotent consumers handle retries safely.
• Task queues (Celery-style) are a close cousin — they run scheduled or computationally heavy jobs and return a result, not just fire-and-forget messages.
• Back pressure caps queue growth: once it's full, reject new work (HTTP 503) instead of drowning in an unbounded backlog.

Common pitfalls

• An unbounded queue can grow past memory, spilling to disk and slowing every job behind it — cap it and apply back pressure.
• Not every workflow benefits from async — cheap, latency-sensitive operations just pay the queue's overhead for nothing.
• Rejected clients should retry with exponential backoff, or a recovering queue gets immediately re-slammed.