A queue moves pressure; it does not remove it
Queues are useful because they separate arrival time from processing time, but they do not make infinite work possible. When producers publish faster than workers can complete, backlog age grows. A queue that keeps accepting work without a bound can turn a short dependency outage into hours of delayed jobs, exhausted storage, and surprising customer behavior. Backpressure is the set of rules that tells producers when to slow down, reject, or defer work.
Start with the business promise for each queue. A thumbnail can wait ten minutes, a webhook acknowledgement cannot, and a password reset may expire before a long backlog completes. Record maximum useful age, retry budget, visibility timeout, expected throughput, and side-effect policy. The queue name should communicate the work and priority so operators can see which backlog is actually harming users.
Bound the producer and the queue
A producer should have a timeout and a clear response when publishing is unavailable or the queue is full. Do not let an HTTP request wait indefinitely for a queue broker. For interactive work, return an accepted status only after the event is durably enqueued. For optional work, drop or coalesce requests according to a documented policy. A queue length cap can protect storage, but it must be paired with a user-facing outcome so dropped work is not mistaken for success.
Use a priority or separate queue when one workload must not starve another. A single FIFO queue can make a slow low-value job block urgent work. Fair scheduling is not free; measure how much capacity each class receives and reserve enough for critical operations. If producers can batch work, publish bounded batches and use a cost field so one large message does not consume an unfair share of a worker.
Scale workers from age, not only length
Queue length alone does not describe customer impact. Ten thousand fast jobs may be healthier than ten old jobs that are waiting behind a stuck dependency. Track oldest message age, enqueue-to-start delay, processing duration, retry count, and completion age. Scale workers based on the signal that matches the service promise. A large worker fleet cannot fix a dependency that is returning errors; it can amplify the outage.
Workers should have a bounded concurrency and a per-dependency budget. If each worker can start ten calls to the same provider, scaling from ten to one hundred workers creates a thousand-call burst. Coordinate worker concurrency with database pools, rate limits, and downstream quotas. Make scale-out and scale-in graceful so a worker does not abandon a message without a visibility or lease rule.
Make retries finite and classifiable
Retry transient failures with exponential backoff and jitter, but do not retry validation errors, revoked credentials, or payloads that will never succeed. Store the attempt count, last failure category, next attempt time, and a safe error summary. A message should have a maximum delivery age or attempt budget. Infinite retries create a permanent backlog and conceal the fact that the business action cannot complete.
A retry policy must understand side effects. The worker may have completed an external call before the connection failed, so the message handler needs an idempotency key or reconciliation step. A visibility timeout should exceed the expected processing time with room for cleanup, but not be so long that a crashed worker hides work for hours. Extend a lease only when progress is real and observable.
Use dead letters as a recovery path
A dead-letter queue is not a trash can. Move exhausted messages with the original ID, attempt history, error category, code version, and timestamps. Protect the payload according to its sensitivity and set a retention policy. Alert on dead-letter age and volume, not just on the first message. A runbook should explain whether an operator can fix the data, wait for a dependency, or intentionally discard the work with an audit record.
Replay should be explicit and bounded. Show the selected messages, current state, expected side effects, and destination queue before starting. Replaying into the same failing path can create a second backlog. Use a shadow or lower-priority queue when validating a handler fix. A safe replay tool makes recovery a controlled change instead of a sequence of shell commands pasted during an incident.
Observe the queue as a user journey
Connect a logical operation ID from the producer to the queue message, worker, downstream calls, and final state. Measure accepted, started, completed, failed, retried, expired, and dead-lettered counts. Break down latency by queue and work type. A successful enqueue is not a successful user action. Product metrics should show whether delayed work still meets its usefulness window.
Avoid logging full payloads in every worker attempt. Use a redacted envelope, message ID, tenant-safe identifier, and error category. Keep a controlled diagnostic view for an operator who needs to inspect one message. The queue is a boundary where data can persist longer than a request, so retention and access control are part of the design.
Load-test the failure, not just the happy path
Generate bursts, slow workers, dependency 429 responses, broker latency, worker crashes, poison messages, and a sudden scale-out. Assert that producer latency stays bounded, critical queues remain fair, backlog age triggers the intended response, and retries do not multiply side effects. Test a full queue and verify the caller receives a meaningful status. Include a recovery test that drains the backlog without overwhelming the dependency.
Healthy backpressure makes an uncomfortable state visible early. It slows or rejects work before storage, memory, or a dependency collapses, while preserving the most valuable operations. Define limits, scale from age, classify retries, quarantine poison messages, and rehearse replay. A queue should buy time for recovery, not hide the need for it.
Make overload a deliberate user outcome
When a queue reaches its safe depth, choose the outcome before the incident: reject new work, shed optional work, slow producers, or route to a lower-cost path. Expose queue age and oldest-message time, not only depth, because a small queue can still be stuck. Test consumer failure, poison messages, provider throttling, and recovery. Backpressure is effective when it protects the dependency and gives the caller a truthful retry or status signal.
Implementation example
Define maximum useful age, retry budget, visibility timeout, concurrency, and dead-letter behavior for each queue. Have producers receive a bounded response when admission is unsafe, and let workers report completion only after the side effect is durable. Separate optional work from user-critical work so one backlog cannot consume all capacity.
if queue.oldest_age > max_useful_age:
reject_or_defer(producer_request)
elif dependency_unhealthy:
reduce_consumer_concurrency()Verify and troubleshoot
Simulate a burst, worker outage, poison message, dependency throttle, full queue, and recovery. Track depth, oldest-message age, throughput, retry count, in-flight work, and dead-letter rate. Confirm that an accepted request has a truthful status path and that the queue cannot grow indefinitely without an alert or admission decision.
Operations and recovery
Set capacity limits and an operator playbook for slowing producers, shedding optional work, increasing workers, or replaying dead letters. Protect database and provider quotas when scaling consumers. If backlog age exceeds the product promise, communicate degraded state and stop accepting work that cannot complete usefully rather than hiding the failure in storage.
References and further reading
Use the selected queue provider's visibility, retry, dead-letter, and quota documentation, plus queueing theory basics. Define whether the queue promises ordering, at-least-once delivery, or best-effort work.