Idempotent background jobs that recover cleanly

Background workers operate in an uncertain world. A process can finish a payment-provider call and crash before recording completion. A queue can redeliver after a visibility timeout. A deploy can interrupt a worker between two statements. The system cannot reliably distinguish every completed job from every abandoned job, so at-least-once delivery is the practical default. Treat duplicate execution as part of the job contract rather than an exceptional bug.

Define the logical job identity separately from the delivery attempt. The logical identity might be an order export, a webhook event, or a report generation request. Each attempt gets its own timestamps and error history, but the durable job record owns the state and idempotency key. A retry should resume or safely repeat the logical operation, not create a second business action.

A common failure occurs when an application writes a database change and then publishes a queue message. The process can commit the database transaction and crash before publishing, leaving work that no worker knows about. The opposite order can publish a message and then fail the database transaction, causing a worker to process an intent that never committed. Use an outbox record in the same transaction as the business change, then publish from the outbox with a retryable dispatcher.

The outbox should contain a stable event ID, job type, payload or reference, created time, attempt state, and a safe delivery history. A dispatcher must claim rows atomically and tolerate duplicate publication. The consumer still needs idempotency because the broker or dispatcher can deliver the message more than once. This two-stage design makes the gap observable instead of pretending a distributed transaction exists.

A job state such as queued, running, succeeded, failed, or dead-lettered is more useful than a single processed flag. A worker claims a queued job with a lease owner and expiry, performs bounded work, and commits a conditional transition. If the process dies, another worker can reclaim an expired lease. A completed job must not move back to running because a late duplicate delivery arrived.

Keep attempt count, last error category, next retry time, and code version with the job. A lease should expire after the normal processing budget plus cleanup time, not after an arbitrary number that hides slow work. If a job needs more time, renew only when progress is measurable. A worker that renews forever while blocked on a provider prevents recovery and turns one bad job into a permanent reservation.

Database uniqueness protects database writes, but a job often calls email, payments, storage, or another API. Pass a stable idempotency key to downstream services when supported. If the provider lacks idempotency, create an outbox entry for the side effect with a unique business key and record the downstream request ID. On retry after a timeout, reconcile the provider or query the downstream result before sending a second request.

Do not mark a job succeeded before its required side effects are durable. Do not call every side effect and then assume a final database update will always commit. Model the workflow so each side effect has a state, an owner, and a retry policy. If compensation is possible, represent compensation as a new action with its own audit trail rather than rewriting the original job history.

A malformed payload, missing customer record, rate limit, network timeout, and provider outage do not deserve the same retry policy. Classify errors as permanent, transient, or unknown. Permanent errors should move to a review or dead-letter state with a useful explanation. Transient errors should use exponential backoff and a maximum age. Unknown errors can retry within a small budget while alerting the owner.

A poison job is one that fails repeatedly without changing the conditions that caused the failure. Detect it with attempt and age limits, not just a count, because a job that retries once an hour can remain harmful for days. Preserve the original payload or a redacted reference, the error timeline, and the handler version. A replay should require an operator to acknowledge the current state and expected side effects.

Measure queue delay, processing duration, success rate, retry rate, lease expirations, age of the oldest job, dead-letter volume, and downstream latency. Break down metrics by job type and priority, not by unbounded customer identifiers. A worker can have healthy CPU while an invoice job is aging past its usefulness window. Alert on that business consequence and link to the job state view.

Trace one logical job across enqueue, claim, downstream calls, and completion. Log IDs and safe categories instead of full payloads. Retain detailed attempts for the period needed to recover or audit, then remove sensitive content. A job record should answer what was intended, who claimed it, what happened, and whether a replay is safe.

Inject failures after the database commit, after message publication, after lease acquisition, after a downstream response, and before completion. Deliver the same message concurrently and verify one logical outcome. Test lease expiration, worker cancellation, provider 429 responses, malformed payloads, dead-letter replay, and a full queue. Use a fake clock and fake downstream service so tests can reproduce ambiguous timeouts without sleeping.

Idempotent jobs are designed around recovery, not around a perfect worker. Persist intent, claim with a lease, make each side effect deduplicated, classify errors, and keep the state machine observable. When a process crashes at the worst possible moment, the next worker should find enough durable evidence to continue safely instead of guessing whether the customer has already been charged.

Store the operation key with the outcome or a durable processing marker in the same transactional boundary as the side effect whenever possible. Define how long the key remains valid, what happens when the payload changes, and how operators replay a failed job safely. Test duplicate delivery, timeout after commit, worker restart, and a retry arriving after a success response. Idempotency turns at-least-once delivery into a predictable contract instead of a source of duplicate charges or messages.

Separate logical job identity from delivery attempts and claim work with a durable state transition. Store the idempotency key with the business result or processing marker, and make retry after timeout safe. For an external side effect, use the provider's idempotency key or an outbox record so the worker can reconcile an uncertain response.

Crash a worker after the side effect and before acknowledgement, redeliver after a visibility timeout, send two concurrent attempts, and retry a permanently invalid payload. Assert one business result, an observable duplicate, bounded attempts, and a clear terminal state. Preserve provider request IDs so an operator can reconcile an ambiguous call instead of guessing.

Monitor queue age, attempt count, lease expiry, terminal failures, and jobs stuck in started state. Provide a safe replay command that requires an owner and reason. If a side effect cannot be reversed, add a compensating action and mark the original outcome honestly; do not delete the job record to make the queue look healthy.

Use the queue provider's delivery semantics, transactional outbox, idempotent consumer, and provider-specific idempotency documentation. State the key lifetime and duplicate outcome in the job contract.