Guided journey

Make Slow Work Reliable

An intermediate journey: move slow work off the request path, add retries with backoff, make jobs idempotent, and catch failures in a dead letter queue.

Intermediate2 weekends9 milestones15 min read
You walk away with

Slow work moved off the request path into a job queue with retries, idempotency, a dead letter queue, and monitoring that notices a stuck worker.

Assume you have

An app that sends email, calls external APIs, or processes uploads inside a request handler, and a place to run a long-lived process.

Stack used

Redis plus a worker process. Postgres-backed queues work too; the patterns are the same.

The project

Take one slow, failure-prone operation out of a request handler — sending a welcome email, resizing an upload, syncing to a third-party API — and move it into a queue that retries, de-duplicates, and reports when it gives up.

By the end you will have: a worker process, a job that survives an outage, an idempotent handler, a dead letter queue inspected at least once, and a heartbeat that tells you a worker has stalled.

Before you start

  • The handler that currently does the slow work, and evidence it is slow — a log line, a timeout, a slow request in monitoring.
  • A place to run a second process: the host’s worker feature, a container, or a small server.
  • Background Jobs and Scheduled Tasks as the map for the whole area.

Milestone — Find the work that does not belong in a request

Build: a list of operations to move, and one to start with.

Not all work should be queued; some of it is cheap and synchronous. Identifying the right candidate is the first decision.

Do it

  1. Look at request timings and find handlers over a second.
  2. Move to the list anything that calls a network, sends a message, or can fail transiently.
  3. Pick the single worst offender to migrate first.

Done when you can name the operation, its current average duration, and what it talks to.

Milestone — Choose queue or scheduler — and say why

Build: a written choice between an event-driven queue and a time-driven schedule.

“Run when this happens” and “run at this time” are different mechanisms with different failure modes. Choosing the wrong one is a rewrite later.

Do it

  1. If the work is triggered by a user action, you want a queue.
  2. If it is periodic by nature, you want a scheduler — see What Is a Cron Job? and note that cron jobs fail silently by default.
  3. Record the choice in NOTES.md, including which retry semantics you need.

Done when the mechanism is named and justified. DLQ vs Retry is the decision guide for the failure-handling half of the choice.

Milestone — Build the worker and enqueue the first job

Build: a producer that enqueues and a worker that consumes, running end to end.

Get one trivial job — the small, observable one — through the whole pipeline before adding retry logic. Debugging a queue and a retry policy at once is twice the work.

Prompt your agent

Move the <slow operation> out of the request handler.

- The handler enqueues a job and returns immediately.
- A separate worker process consumes jobs and performs
  the work.
- The queue is Redis-backed with an at-least-once delivery
  guarantee.
- Log the job id at enqueue and at completion, so a job
  can be traced end to end.

Show me the producer code and the worker entrypoint.

Done when the request returns fast, the worker performs the operation, and you can follow one job id from enqueue to completion in the logs. How to Build a Background Job Queue is the step-by-step.

Milestone — Make the job idempotent

Build: a handler that produces the same result whether it runs once or three times.

At-least-once delivery means a job will run twice eventually. Design for that now, not after the duplicate emails.

Do it

  1. Give each job a stable id derived from the operation, not a random one.
  2. Before doing the work, check whether it has already been done — a status column, a processed-ids set, or a natural key.
  3. Test it: run the same job id twice and confirm exactly one side effect.

Done when a forced duplicate produces one result. What Is Idempotency? covers the patterns, including the one for webhooks.

Milestone — Add retries with backoff and jitter

Build: automatic retries for transient failures, with a cap.

Retrying instantly turns a wobbling dependency into an outage. Backoff with jitter gives the far end time to recover and stops your workers from synchronising their retries.

Do it

  1. Classify failures: network timeouts and 5xx are retryable; validation errors are not.
  2. Retry with exponential backoff plus jitter, capped at a few attempts.
  3. Log each attempt with its number and reason.

Done when a dependency that fails twice then succeeds results in a completed job, and a permanently failing one stops after the cap. What Is Exponential Backoff? and How to Add Retry Logic to API Calls have the implementation.

Milestone — Catch what cannot be recovered

Build: a dead letter queue, with an alert and a workflow for reading it.

Retries end somewhere. Without a dead letter queue, exhausted jobs vanish, which is the same as silent data loss.

Do it

  1. Route jobs that exhaust retries to a dead letter queue with the original payload and the error.
  2. Alert when anything lands there — this is a “humans must look” signal, not noise.
  3. Inspect one entry and write down the steps to replay or discard it.

Done when you have seen a job arrive in the DLQ, read the failure, and either fixed and replayed it or discarded it deliberately. What Is a Dead Letter Queue? is the reference.

Milestone — Shut down without losing a job

Build: a worker that finishes its current job and exits cleanly on SIGTERM.

Workers get restarted on every deploy. A killed worker mid-job is a lost or duplicated job, depending on whether you got the idempotency right.

Do it

  1. Handle SIGTERM: stop accepting new jobs, finish the in-flight one, then exit within the platform’s grace period.
  2. Make sure the job is only acknowledged after it completes, so a crash returns it to the queue.
  3. Test by sending SIGTERM while a job is running.

Done when a deploy never drops an in-flight job. See What Is Graceful Shutdown? and Why Does My App Ignore SIGTERM?.

Milestone — Notice when a worker stops working

Build: a heartbeat for scheduled work and a queue-depth alert for queues.

A dead worker looks exactly like a quiet period. If nothing watches the pipeline, you discover the backlog when a user asks why their email never arrived.

Do it

  1. For scheduled jobs, add a heartbeat that alerts on a missed run — How to Monitor Your Cron Jobs.
  2. For queues, alert on depth or oldest-job age, not just on errors.
  3. Trigger both alerts on purpose once.

Done when you have received a real “job did not run” alert. What Is a Health Check? covers the endpoint to point monitoring at.

Milestone — Protect the systems you call

Build: a rate limit and a circuit breaker around external APIs.

Your queue can become a denial-of-service tool pointed at a partner if it retries without limits. Bound the outbound pressure.

Do it

  1. Limit concurrent jobs per external service.
  2. Add a circuit breaker so a failing dependency stops receiving traffic for a cooldown period.
  3. Store timestamps in UTC so schedules and logs agree across machines — Why Should Servers Always Use UTC?.

Done when a dependency outage does not turn into a retry storm in your logs. See What Is the Circuit Breaker Pattern?.

Ship checklist

  • Slow, network-touching work runs in a worker, not a request handler.
  • Queue versus scheduler was chosen deliberately.
  • Every job carries a traceable id.
  • Job handlers are idempotent, verified by running one twice.
  • Retries use backoff with jitter and a bounded attempt count.
  • Exhausted jobs land in a dead letter queue with an alert.
  • Workers shut down gracefully and acknowledge only completed jobs.
  • Scheduled work has a heartbeat; queues have a depth alert.
  • Outbound calls are rate limited and can be tripped by a circuit breaker.

Where this journey goes wrong

  • Queueing everything. Synchronous, cheap work belongs in the request.
  • Retrying validation errors. They fail forever and waste the queue.
  • Assuming exactly-once delivery. At-least-once is the real guarantee; idempotency absorbs the difference.
  • A dead letter queue nobody reads. It becomes a graveyard, not a signal.
  • No heartbeat. A stopped worker is invisible until a user complains.

FAQ

When is a queue overkill?

When the work is fast, local, and cannot fail — a template render or a synchronous validation. A queue adds a process, a broker, and a new class of bug. Introduce one when work is slow (over a second), touches a network, or must be retried.

Why does idempotency matter if I have retries?

Because retries make duplicate execution normal. Any job that can run twice — after a timeout, a crash, or an at-least-once queue — must produce the same result the second time. Otherwise a transient failure becomes a double charge or a duplicate email.

Share: