Background Jobs Are Where Systems Actually Break
Queues, retries, and at-least-once delivery are unglamorous, but they're where most of the real failures in a production system actually live
When something goes wrong in a web app, most people picture a broken button or a 500 page. Those are the visible failures — easy to notice, easy to reproduce, usually easy to fix. The failures I actually worry about live somewhere nobody's looking: in the crons, the queues, the retry logic that runs at 3am with no one watching.
Background jobs get less scrutiny than user-facing code for an understandable reason — nobody's staring at them when they run. A broken checkout button gets reported in minutes. A cron job that's been silently failing for three weeks gets reported when someone notices the thing it was supposed to do never happened, and by then the data's already wrong in ways that are hard to reconstruct.
At-least-once is the default, not an edge case
Most queueing systems promise "at-least-once" delivery, not "exactly-once." That's not a limitation someone forgot to fix — it's the honest tradeoff. Guaranteeing a message is processed exactly one time, across network failures and process crashes, is expensive enough that almost nobody actually does it. What you get instead is a guarantee that a job will run, plus the possibility that it runs twice.
Most background job code I've reviewed doesn't act like it knows this. It sends the email, charges the card, or deletes the row as if the function is only ever going to be called once, ever. Then a worker times out mid-run, the queue redelivers, and the same job fires again against a slightly different world. If the job wasn't written to expect that, the second run isn't a no-op — it's a second email, a second charge, a delete against rows that already changed underneath it.
The fix isn't exotic. It's writing every job as if it might be handed the same task twice and making sure the second run either does nothing or produces the same result as the first. That habit is cheap to build in and expensive to retrofit after the job has been running in production for a year.
The clock you inherited isn't the clock you need
A lot of background-job bugs are really timeout bugs wearing a disguise. Database clients, HTTP libraries, and transaction wrappers all ship with default timeouts tuned for interactive requests — a few seconds, because a human is waiting. Background jobs don't have a human waiting. They batch-process, they touch thousands of rows, they call slow external APIs. Wrap that work in a transaction with the library default and it'll work fine in every test, then fail intermittently in production the first time the batch is large enough to actually take the time it needs.
The failure mode here is nasty because it's load-dependent. It passes every test you write against a small dataset and only breaks once real volume shows up — usually right when you can least afford a mystery.
No deadline means it never ends
The opposite problem is just as common: a job or a process that waits with no deadline at all. It calls out to something — a subprocess, a webhook, another service — and just... waits. If that dependency dies cleanly, fine, the wait ends. If it dies silently, or the network drops the connection without a proper close, the job sits there forever, holding a worker slot, looking alive to any monitor that only checks "is the process running" rather than "is it making progress."
I've seen this pattern more than once in automated build and review pipelines I've worked on — a polling loop waiting on output from a background command, with no upper bound on how long it's willing to wait. The child process disappears; the loop doesn't notice, because nothing ever told it to notice. The run parks forever in a "still working" state that nobody escalates, because nothing is technically broken. It's just stuck.
Every wait needs an edge. Not just "will this eventually respond" but "what happens if it never does" — and that second question has to have an answer that isn't "someone finds it manually next week."
Concurrency finds the race you didn't test for
Background jobs also tend to run more than one instance at a time, which surfaces bugs that a single-threaded mental model never catches. A job that writes to a temp file at a fixed path works fine when you run it once. Run two instances against the same input at the same moment — which will happen, because that's what queues are for — and now they're both writing to and cleaning up the same path, and one instance's cleanup deletes the other instance's work mid-flight.
This is the kind of bug that's invisible in development, rare in staging, and eventually shows up in production as an intermittent, unreproducible failure that looks like corruption. The fix is almost always the same: give every concurrent run its own namespace — a unique suffix on the temp path, a row-level lock, anything that stops two workers from stepping on the same resource. It's a small fix. The hard part is remembering to ask the question before the collision happens instead of after.
Treat every job like it doesn't trust the world
The common thread across all of this is that background jobs run in conditions nobody designs for on the first pass: retried, delayed, run in parallel, handed a clock that doesn't match the work. A job that assumes a clean, single, well-timed execution is a job that will eventually be wrong, and the wrongness will show up as data drift or duplicate side effects rather than a clean error message pointing at the cause.
None of this is exciting work. Nobody demos a retry-safe cron job. But it's the layer where a system's actual reliability gets decided, long after the parts anyone reviews carefully have shipped.