What Is a Distributed Lock?

Quick answer
A distributed lock is a mutual-exclusion mechanism shared by processes that do not share memory, typically implemented with a shared store such as Redis, a database, or a coordination service.
The basic primitive is set-if-not-exists with a time-to-live, so exactly one holder can acquire it and a crashed holder does not block the resource forever.
The hard part is that a lock holder can stall — a pause, a network partition — and continue after its lock expired while another process also holds it.
Fencing tokens are the mitigation: each acquisition gets an increasing number, and the resource rejects writes from an older token.
Often the better answer is to make the operation idempotent or to use a database constraint instead of a lock, because correctness from locking alone is fragile.

What is a distributed lock?

It is a way to ensure that at most one process is doing a particular thing at a time across machines. Within one process you would use a mutex; across several servers there is no shared memory, so the lock has to live in a shared service both processes can reach. The shared service holds a key that represents the lock, and only the holder may proceed.

The typical implementation is an atomic set-if-not-exists with an expiry: an acquisition that finds the key absent sets it and succeeds, one that finds it present fails, and the expiry removes a lock whose holder died. That is the whole mechanism, and it is where most explanations stop. The interesting part is everything the mechanism does not guarantee.

The two failure modes that matter

The first is expiry while still holding. A holder acquires a lock for thirty seconds, then stalls for longer than that — a stop-the-world pause, a slow disk, a network partition. The lock expires, another process acquires it, and now two processes believe they hold the same lock. Any correctness argument that relied on mutual exclusion is void, and the failure appears as an intermittent double-processing that is impossible to reproduce.

The second is locking the wrong scope. A lock protects a resource only if every writer also takes the lock. If one code path updates the row directly and another takes the lock first, the lock has excluded nothing. This is the same property as cache invalidation done in one path out of three: it looks like a control and behaves like a suggestion.

What makes it safer

Fencing tokens address the first failure. Each successful acquisition is issued a monotonically increasing number, and the protected resource is told to reject operations carrying a token lower than the highest it has seen. A holder that stalls and resumes finds its token stale and its write refused, so the second holder’s work is not corrupted. This requires the resource to cooperate, which is why it is often implemented inside the database — a conditional update with a version check — rather than around it.

Short expiries with periodic renewal reduce the window; a lease that is renewed while alive and allowed to lapse on death is the standard shape. Finally, treat the lock as an optimisation rather than a guarantee where correctness is at stake: the operation should still be safe if it runs twice, and the lock only makes it unlikely rather than impossible. That framing changes the design in the right direction.

# A lease-based lock with a token, used with a conditional write.
import time, uuid

def acquire(cache, name, ttl=30):
    token = f'{int(time.time()*1000)}-{uuid.uuid4()}'  # monotonic-ish + unique
    # SET NX with expiry: atomic, and self-healing if the holder dies.
    if cache.set(f'lock:{name}', token, nx=True, ex=ttl):
        return token
    return None

def renew(cache, name, token, ttl=30):
    # Only the current holder may extend it.
    if cache.get(f'lock:{name}') == token:
        cache.expire(f'lock:{name}', ttl)
        return True
    return False

# The protected write is conditional on the token in the database,
# so a stale holder is refused even if it resumes after expiry:
# UPDATE jobs SET status='running', fence = :token
#  WHERE id = :id AND fence < :token;

Alternatives worth checking first

A database constraint is often enough. A unique index on the field that must not duplicate enforces the invariant no matter how many processes try, and it cannot expire or be forgotten. A conditional update with a version column gives compare-and-swap semantics for state transitions, which covers most scheduler and job-claiming needs. Both use machinery you already run and are correct under failure in ways a bespoke lock is not.

Where the shared store is the right primitive, prefer an established implementation over a hand-rolled one. Redis’s own documentation describes the pitfalls in detail, and coordination services built for this purpose provide leases and sessions as first-class features. Writing a lock from scratch is a reasonable exercise and a poor production choice, because the failure modes are subtle and only appear under load.

Where this bites vibecoders
Generated code that needs “only one worker should do this” reaches for a lock, often without an expiry, or with one that is longer than the work. The result works in testing and produces duplicate processing in production during a pause. Before accepting a lock, ask what happens if the holder stalls past the expiry, and whether the operation is safe if it runs twice — if the second answer is no, the lock is carrying weight it cannot bear.

Where AI coding assistants get this wrong

Acquiring a lock with no expiry, so a crashed holder blocks the resource forever.
Setting an expiry shorter than the work, allowing two holders.
Protecting a resource with a lock that not every writer takes.
Never renewing the lease, so long operations expire while running.
Implementing waiting with a tight retry loop that hammers the lock store.
Using a lock where a unique constraint or a conditional update would be correct.

Checklist

Decide whether the operation is safe to run twice, honestly.
Check whether a unique constraint or conditional update removes the need for a lock.
If a lock is needed, give it a short expiry and renew it while alive.
Protect the write with a fencing token or a version check in the store.
Bound the wait for the lock, and decide what happens when it is not acquired.
Test the failure path: kill the holder mid-operation and confirm recovery.

FAQ

Is a Redis lock safe?

It is safe for the purpose of reducing concurrent work, and not safe as the sole guarantee of correctness. Expiry while holding is possible, so a stalled holder can act after another process acquired the lock. Used with a fencing token or a conditional write, it becomes sound; used alone, it depends on the stall never happening, which is not a guarantee.

What is a fencing token?

A number that increases with every lock acquisition, passed to the protected resource so it can reject operations from an older holder. If a stalled holder resumes with token 7 while the current holder has token 8, the resource refuses the write. It is the mechanism that turns a lease into actual safety.

Do I need a distributed lock at all?

Much less often than it seems. Scheduled work can be claimed by a conditional database update. Duplicate records can be prevented by a unique constraint. State transitions can use compare-and-swap. A lock is justified when the work is genuinely exclusive, expensive, and not naturally idempotent — and even then, making the work idempotent is usually the better investment.

How should a waiter behave when the lock is held?

Bounded and backing off: wait a short interval, retry a few times, and then either skip the work or return a clear result. A tight retry loop turns the lock store into the bottleneck. Where possible, have the next run pick the work up later rather than waiting for it, which removes the coordination from the hot path entirely.