Prefer a constraint or queue when it fits
A distributed lock coordinates work across processes, but it is not the first answer to every race. A unique constraint can prevent duplicate ownership, a queue can serialize a task, and an atomic database update can move a state machine without a lock. These primitives carry clearer durability and recovery semantics. Start by writing the invariant and checking whether the database or queue can enforce it directly. A lock adds a failure mode that deserves a strong reason.
Use a lock when several processes must perform a mutually exclusive operation that cannot be represented as one durable state transition. Examples include a scheduled singleton, a cache rebuild, or a migration coordinator. Define what the lock protects, how long work may hold it, and what happens when the owner disappears. If the answer is only that two workers should not run at once, a queue or idempotent operation may be simpler.
Use leases, not permanent ownership
A lease expires after a bounded duration, allowing another worker to recover work after a crash. Store an owner token and expiry together, and release the lease only if the token still matches. A process must not delete a lock it no longer owns because its lease expired and another worker acquired it. The compare-and-delete operation should be atomic in the lock store.
Choose the lease duration from the maximum safe work interval plus a renewal margin. A long lease delays recovery; a short lease increases renewal traffic and the chance that valid work loses ownership. Renewal should require observable progress and stop when cancellation begins. Do not renew forever while a downstream provider is unavailable. A lease is a recovery mechanism, not permission to hold a broken workflow indefinitely.
Fencing tokens protect slow owners
A lease can expire while the original owner is paused by a runtime stall or network partition. A second worker acquires the lock and starts fresh work, but the first worker may resume and continue writing. An owner token alone does not stop it if the downstream system only checks that some lock exists. A fencing token is a monotonically increasing number issued on acquisition and included in every protected write. The downstream resource rejects writes with an older token.
Fencing requires cooperation from the resource being protected. A database row can store the latest token and update only when the incoming token is greater or equal. A file or provider that cannot validate the token cannot receive a strong lock guarantee. In that case, make the operation idempotent and use reconciliation rather than claiming that the lock prevents all duplicates.
Avoid lock ordering deadlocks
If a workflow needs more than one lock, define a global order and acquire locks in that order. Two workers that acquire A then B and B then A can deadlock even if each lock implementation is correct. Use a bounded wait and release already acquired locks when the next lock cannot be obtained. Prefer a single composite lock key when the business invariant allows it. A broader lock may reduce concurrency, but it can be easier to recover than a graph of partial ownership.
Do not hold a distributed lock while waiting for an unbounded network call. If the protected work can be represented as a durable state transition, record ownership, release the lock, and let a worker process the state with a lease. If the external call must be serialized, use a queue partition or provider idempotency rather than relying on a process staying healthy for the entire call.
Understand clock and store assumptions
Lease expiry depends on time, but clients and stores can disagree about the current clock. Prefer the lock service's time or a monotonic interval where the protocol supports it. Do not compare a local wall clock to an expiry returned by another machine without accounting for drift. Network latency can consume a meaningful part of a short lease, so measure acquisition and renewal latency.
A lock store outage has to produce a deliberate decision. Failing closed protects exclusivity but can pause work; failing open preserves availability but can duplicate side effects. The right choice depends on the invariant. For a cache rebuild, duplicate work may be acceptable. For a bank transfer, it is not. Document the degraded mode and alert when the system enters it.
Expose ownership without exposing secrets
Operators need to see which operation holds a lock, when it was acquired, its expiry, renewal health, and the associated job or deployment. Store a safe owner ID and operation name, not a credential or full payload. Make lock inspection read-only and restrict it to trusted operators. A manual force-release action should require an explicit resource key, show the expiry and owner, and record the actor and reason.
A lock that has no observable owner becomes an outage during the first contention incident. Emit metrics for acquisition success, wait time, expiry, renewal failure, forced release, and work completed after lease loss. These signals help distinguish a real conflict from a store latency problem.
Test pauses, partitions, and recovery
Test a worker crash before and after acquiring the lease, a pause longer than the lease, a delayed release, concurrent acquisitions, store failover, clock skew, and a downstream write from an expired owner. Assert that fencing rejects stale work and that the new owner can complete. Test a forced release with an active worker and verify that the runbook explains the risk before an operator clicks it.
Distributed locks are useful when their failure semantics are explicit. Prefer constraints and queues, use short leases with atomic ownership checks, fence slow owners, order multiple locks consistently, and keep the protected work observable. The goal is not to make a distributed system look single-threaded. It is to ensure that a delayed or disconnected process cannot silently claim authority it no longer owns.
Use fencing when a stale holder can write
A lock timeout does not stop a paused process from resuming and writing after another worker takes the lock. When stale work can damage state, issue a monotonically increasing fencing token and have the storage layer reject older tokens. Test lease expiry, clock pauses, process termination, network partitions, and lock-service recovery. The lock should coordinate work, while the protected system independently verifies that the caller still has authority to commit.
Implementation example
Use a lock only when a durable constraint, queue, or atomic state transition cannot enforce the invariant. Give the lease an owner, expiry, renewal policy, and fencing token. The protected database or API must reject stale tokens so a paused process cannot continue writing after its lease is replaced.
token = lock_service.acquire(resource)
write_state(resource, value, fencing_token=token)
lock_service.release(resource, token)Verify and troubleshoot
Test owner pause, lease expiry, network partition, lock-service restart, competing workers, clock skew, and a process that resumes after a long stop. Assert mutual exclusion, bounded wait, stale-owner rejection, and recovery after the holder dies. Instrument acquisition wait, lease age, renew failures, and fencing conflicts.
Operations and recovery
Keep lock duration shorter than the protected operation's safe window or use fencing to make overrun harmless. Alert on orphaned locks and repeated acquisition failures. If the lock service is unavailable, fail closed for destructive work and prefer idempotent retries. Never delete a lock manually without checking whether the owner is still committing state.
References and further reading
Compare database constraints, queue serialization, lease-based locks, and fencing-token patterns. Treat the lock service's consistency and failure semantics as part of the design, not an implementation detail.