Background Jobs and Queues

Not all work a backend does needs to happen before it responds to a request. Sending a confirmation email, resizing an uploaded image, generating a report — none of these need to finish before the user sees "success." Doing them inline makes the request slower than it needs to be, and ties the request's success to the success of work that isn't actually essential to it.

The problem with doing everything inline

// Inline — the request waits on all of this POST /signup → create user in database → send welcome email (slow: network call to email provider) → generate onboarding PDF (slow: CPU-bound work) → 201 Created (user has been waiting this whole time)

If the email provider is slow or briefly down, the entire signup fails or hangs — even though account creation itself succeeded. The user's experience is now hostage to a dependency that has nothing to do with whether their account exists.

Moving work to the background

A background job is work handed off to run outside the request/response cycle, typically via a queue — the request pushes a small message describing the work, and a separate worker process picks it up and executes it independently.

POST /signup → create user in database → push "send_welcome_email" job to queue → 201 Created (returns immediately) Worker process (running separately, continuously): → pulls "send_welcome_email" job from queue → sends the email → on failure: retries with backoff, or moves to a dead-letter queue

Common queue technologies include Redis-backed queues (BullMQ, Sidekiq), RabbitMQ, and cloud-managed options like AWS SQS — the request-handling code just needs to push a small, serializable message, not know anything about how or when it gets processed.

Why decoupling this way matters

• Faster responses — the client gets a response as soon as the essential work is done, not after every side effect finishes too.
• Resilience — if the email provider is down, the job just retries later; it doesn't fail the signup itself.
• Load smoothing — a sudden spike of 10,000 signups doesn't mean 10,000 simultaneous emails; workers process the queue at a sustainable, controlled rate.
• Independent scaling — you can run more or fewer workers based on queue depth, separately from how many web servers you run.

Retries and dead-letter queues

Background jobs need explicit failure handling, since there's no user staring at a spinner to notice something went wrong. A job that fails is typically retried automatically, often with exponential backoff (wait longer between each attempt). After enough failed attempts, it's moved to a dead-letter queue — a holding area for jobs that consistently fail, so a team can investigate rather than losing the job silently or retrying it forever.

What belongs in a request vs. a background job

The dividing line is usually: does the user need this to happen before they can proceed, and can it be confirmed quickly and reliably? Creating the user record and confirming the order both need to happen inline — the user genuinely needs that confirmation. Sending a receipt email, updating an analytics pipeline, or generating a thumbnail can all safely happen a few seconds (or minutes) later.

With the application layer covered, the final section turns to getting a backend running reliably outside a developer's own machine — starting with how a system handles more traffic than one server can.