What Is a Message Queue (and When Do You Actually Need One)?

Quick answer

• A message queue is a store-and-forward buffer between the code that produces work and the code that does it.
• Its real value is decoupling: the request returns immediately, and work survives restarts, deploys, and traffic spikes.
• A database table polled by a worker is often the right answer for a small app — one fewer service to run, debug, and pay for.
• With at-least-once delivery, jobs must be idempotent, and failures need a dead letter queue, or work is silently lost.

What is a message queue?

A message queue is a service that accepts messages, stores them durably, and hands them to consumers — either immediately if a consumer is connected, or later when one becomes available. The producer publishes a message and moves on without knowing or caring who handles it or when.

That independence is the entire point. Without a queue, the code that accepts a request also does the work, so the two scale together, fail together, and both are limited by the slowest part. With a queue, the front end absorbs bursts and returns immediately, and the workers scale independently to match the actual work volume.

The concept is the foundation of Background Jobs and Scheduled Tasks: once slow or failure-prone work is moved off the request path, you need somewhere to put it and something that reliably picks it up.

Where this bites vibecoders

The specific way this bites vibecoders is unrequested infrastructure. Ask for “process this in the background” and the answer is frequently Redis, RabbitMQ, and a separate worker service — three moving parts and a second deployment to keep alive, introduced before the app has a single user. The intent is right and the scale is wrong, and the result is a system that is harder to debug than the blocking version it replaced.

When do you actually need one?

Use a queue when at least one of these is true:

• Work must survive a restart. A deploy kills in-process work; a queue does not.
• The work is slow and the user should not wait. Email, PDF generation, video processing, large imports.
• Traffic is spiky. A queue absorbs a burst and the workers drain it at a steady rate.
• Several services need the same event. Publishing one message can fan out to independent consumers.
• The producer and consumer are separately deployed or separately owned.

Do not add one yet if the work takes milliseconds, runs once at deploy time, or you cannot yet describe what happens when a job fails. That last one matters: a queue converts a visible, loud failure into a silent, deferred one, and you need What Is a Dead Letter Queue? before you convert it.

How do RabbitMQ, SQS, and Kafka differ?

They are usually compared as if they were three versions of the same thing. They are not — they are two different categories with a third in between.

Pick SQS if you want a queue with no infrastructure to run and a small number of background jobs. It is the fastest route to correct behaviour and the one to start with on AWS.

Pick RabbitMQ when messages need routing rules — different consumers for different event types, priority queues, or per-message acknowledgements.

Pick Kafka when you are building an event stream: many consumers each needing the full history in order, replay, or throughput in the hundreds of thousands of messages per second. It is significant infrastructure and almost never the right first choice.

A database table remains a legitimate fourth option, covered in How to Build a Background Job Queue: a jobs table with a status column, a worker polling for pending rows, and a claim query that prevents two workers taking the same job. You can migrate to a broker later, because the interface — enqueue, claim, complete, fail — is the same either way.

How do you handle failures in a queue?

Failures are where queued systems are won or lost, and the rules are specific.

Make jobs idempotent. Most queues deliver at least once, so a job can run twice — a worker can complete the work and crash before recording that it did. Design the job so running it twice is harmless: an upsert instead of an insert, a unique key per job, a check-then-act guarded by a transaction.

Retry with backoff, then dead-letter. Retry immediately and a poisoned job spins forever. Exponential backoff with a cap, then move it to a dead letter queue after a few attempts. The comparison of when to use each is covered in Dead Letter Queue vs Retry.

Monitor the queue, not just the app. The most common silent failure is a growing backlog: workers are crashing, or one message at the head is blocking the partition, and the only symptom is that results arrive late. Alert on oldest-message age and depth, as covered in How to Monitor Your Cron Jobs.

Make work visible. A job that fails in a queue produces no error page and no stack trace. Write the outcome somewhere you will actually look — logs with a job ID, a status column, or a metrics counter.

Where AI coding assistants get this wrong

• Adding a broker and worker service to a small app when a table and a polled worker would do, tripling the deployment surface.
• Enqueuing work after the transaction commits, with no ordering guarantee, so a user sees confirmation for a job that never ran.
• No idempotency key, so a retried job double-charges a card or sends a second email.
• Infinite retries with no dead letter queue, so one bad message blocks the queue forever.
• A visibility timeout shorter than the job’s real runtime, so two workers process the same long job concurrently.
• Failing to update the app’s mental model. A framework’s built-in background task feature is often the correct answer and is skipped entirely.
• Treating a webhook as a queue without the delivery guarantees, retries, or ordering that a real queue provides — see What Is a Webhook?.

Checklist

• Confirm the job must be asynchronous before adding a broker; a table is often enough.
• Define what happens when a job fails before you move work off the request path.
• Make every job idempotent, keyed by a unique identifier.
• Retry with exponential backoff, capped, then dead-letter.
• Set the visibility timeout or ack timeout above the job’s realistic worst-case duration.
• Alert on queue depth and on the age of the oldest pending message.
• Log a job ID through enqueue, start, and completion so a failure is traceable.
• Only reach for Kafka when you genuinely need replay or many independent consumers.
• Keep the enqueue interface the same as your table-based implementation, so migrating later is a change of backend, not of call sites.

FAQ

What is a message queue in simple terms?

A message queue is a middleman that stores messages so the program that sends them does not have to wait for the program that handles them. The sender drops a job into the queue and returns; a worker, running separately, picks jobs up and processes them whenever it can.

Do I need a message queue for a small app?

Usually not yet. A database table with a status column and a worker that polls it does the same job with one fewer piece of infrastructure, and is easier to debug when something is lost. Reach for a real queue when work must survive a deploy, when several workers need the same jobs, or when producers and consumers are separate services.

What is the difference between a queue and a log like Kafka?

A queue delivers each message to one consumer and typically deletes it afterwards, like handing a task to a single person. A log retains messages in ordered partitions that many consumer groups can each read independently, like a bulletin board. Kafka is built for high-throughput event streams; a traditional queue is built for distributing work.

What happens if a message fails to process?

That is the design question a queue forces you to answer, and answering it badly loses work. The usual pattern is to retry with backoff, then move the message to a dead letter queue after a limit so it stops blocking the queue and becomes something you can inspect. The rule is that a job should be idempotent, because at-least-once delivery means it may run twice.

Can a message queue deliver a message twice?

Yes. Most queues guarantee at-least-once rather than exactly-once delivery, because guaranteeing exactly-once across a network boundary is not practical. A worker can finish the work and crash before recording the completion, so the message is redelivered. That is why idempotent job design is not optional.

Is SQS or RabbitMQ better for a beginner?

For a first background-job system, SQS, because there is no server to run, patch, or monitor. Choose RabbitMQ when you need routing rules and multiple exchanges, and Kafka only when you need replay or very high throughput. The important part is the interface you enqueue through, not the broker.

Related topics

• How to Build a Background Job Queue
• What Is a Dead Letter Queue?
• Dead Letter Queue vs Retry: When to Use Each
• What Is a Webhook?
• How to Add Retry Logic to API Calls

Sources

• AWS — What is Amazon SQS?
• RabbitMQ documentation — Queues
• Apache Kafka documentation — Introduction
• Confluent — What is a message broker