How to Build a Background Job Queue

Quick answer
A job queue moves slow work (emails, image resizing, LLM calls) out of the request path so pages respond instantly.
You need three pieces: a place to store jobs, a worker process that runs them, and a way to track results.
Start with a managed or simple tool (Redis + RQ, BullMQ, or a cloud queue) before building anything custom.

When should I use a background job queue?

Whenever a request has to wait on work the user doesn’t need synchronously: sending email, resizing images, calling an LLM, syncing with third parties. Doing this in the request handler means slow pages, timeouts, and lost work when the process restarts. If a task takes more than a few seconds or can fail and be retried, it belongs in a queue.

What does a minimal queue look like?

The canonical small setup is Redis as the queue store, a library like RQ or BullMQ to enqueue and run jobs, and a separate worker process. The web app enqueues a job and returns immediately; the worker picks it up, runs the function, and records the result. The example shows the web side and the worker side for Python + RQ.

# web side: enqueue and return immediately
from redis import Redis
from rq import Queue

q = Queue(connection=Redis())

def send_welcome_email(user_id: str):
    ...  # slow work

@app.post("/signup")
def signup():
    ...
    q.enqueue(send_welcome_email, user_id)
    return {"ok": True}  # responds in milliseconds, email happens later

What does ‘it worked’ look like?

Start the worker with rq worker in a terminal, then hit the endpoint and watch the worker log pick up and complete the job. The HTTP response returns immediately instead of waiting for the slow work. Check the queue dashboard (rq-dashboard) to see job status: queued, in progress, finished, or failed.

# Run the worker (keep it running; deploy it as its own process)
python3 -m rq.worker
# 20:12:10 default: Job OK (send_welcome_email)

Where this bites vibecoders
Vibecoders hit this wall fast: an AI-generated endpoint that calls an LLM or sends mail takes 30 seconds and the browser times out. The assistant’s first fix is often to ‘optimize’ the code or increase timeouts — treating a structural problem as a tuning problem. A queue is the structural fix: the request returns instantly, the work runs separately, and failures become retryable instead of lost.

Where AI coding assistants get this wrong

Running slow work inline in request handlers and calling it ‘good enough’.
Building a custom queue with a database table but no retry, dedup, or DLQ logic.
Enqueueing jobs that reference code the worker process can’t import.
Forgetting that workers need the same dependencies and environment as the web app.

Checklist

Enqueue anything slow or retryable; keep request handlers fast.
Run the worker as a separate process with the same code version as the web app.
Add retries with backoff and a dead letter queue for permanent failures.
Monitor queue depth and worker health like any other service.

FAQ

Do I need a queue if I only have one slow task?

Yes if the task is slow enough to time out a request or you need retries. Even one slow task justifies a queue; the alternative is users staring at spinners and requests failing at the platform’s timeout.

Should I use Redis for the queue or a cloud queue like SQS?

Both work. Redis + RQ or BullMQ is simpler to run locally and reason about; SQS removes the need to run Redis and scales without ops. If you already run Redis for caching, starting with it is the pragmatic choice.