Distributed Locks

A distributed lock lets separate processes agree on which one currently has permission to run a protected operation. They ask an external authority, such as a database, ZooKeeper, etcd or Redis, which grants ownership for a limited time. A lock reduces conflicting work, but holding it does not stop a process that has paused or lost contact from acting after its permission ran out. For that, the resource being modified must also refuse stale actors, through a fencing token, a unique constraint or a conditional write.

Take a billing service with three instances. Every minute all three notice that invoice inv_72 is due. If each one generates and sends it, the customer receives three invoices or three charges. Having the instances compete for a lock named invoice:inv_72, and letting only the winner proceed, is a sound first step. It turns three attempts into one in the normal case, which is most of the value.

The mental model is three parties with different jobs. Workers want to do the work. The lock authority decides who may try, and for how long. The protected resource is whatever actually changes: a row, an object, a payment provider, a device. The authority and the resource are almost always different systems, so the authority's decision can be out of date by the time it reaches the resource. Everything difficult about distributed locks follows from that gap, and the second half of this lesson is about closing it at the resource.

The same lease-plus-generation idea sits underneath job processing, where a worker owns a job row for a limited time. This lesson owns the underlying mechanics: leases, owner tokens, fencing and what each coordination service really promises, and that lesson applies them to claiming rows.

A mutex inside Worker A stops other threads of Worker A from entering the critical section. Worker B, on another host, has its own mutex, and neither can observe the other. In-process locking works because threads share memory and fail together: if the process dies, every thread and every lock in it dies too. Neither property survives across machines. Participants can crash independently, and a process can stay alive while being unable to reach the lock service, or the resource, or both.

So a shared authority becomes the common decision point. A correct design names four things: the lock key (what is being protected), the authority that grants it, the maximum validity of a grant, and the protected write that has to reject stale actors. Teams usually get the first three right and forget the fourth. Without it the lock is advisory: it works as long as every actor behaves and nobody is delayed.

The drawing shows where each decision is made. The authority is consulted once, at acquisition, and never sees the data. Every later write goes straight from the worker to the resource, so the resource is the only party that can atomically say no. A lock decides who may try; the resource decides which writes count.

Why this is not just the classic read-modify-write race

Two servers each read stock = 1, both decide the item is available, and both write stock = 0: one unit has been sold twice. A lock around the read-modify-write fixes it, but so does collapsing the check and the update into one statement the database executes atomically. That alternative is shown in the database section and is usually the better fix, since it has no lock service to fail.

A lease-based lock has four steps: acquire, obtain an owner token, execute while the lease is valid, and release only if still the owner. The lease is what makes the lock survivable. A holder that crashes without releasing cannot block everyone forever, because the grant expires by itself. The price is that ownership can end without the owner noticing, which is the subject of the next section.

Acquisition with Redis

In Redis the acquisition is a single command, SET lock:invoice:72 <token> NX PX 30000. NX makes the write conditional on the key not existing, and PX attaches an expiry in milliseconds. It has to be one command. The older two-step form, SETNX followed by EXPIRE, can leave an immortal lock behind if the process dies between the two calls. The value is a random token unique to the caller (at least 128 bits of randomness, not a hostname or a PID that can repeat), and it is what lets the owner prove later that the key is still its own.

Release must compare before it deletes

An unconditional DEL is a bug waiting for a slow day. Worker A's lease expires while A is stalled. Worker B acquires the same key. A wakes up, finishes, and deletes the key, which now holds B's token. Worker C then acquires it, and B and C are both inside the critical section.

The release must read the value, compare it with the caller's token, and delete in one atomic step. In Redis that means a short server-side script, because a separate GET then DEL leaves a window between them. Some newer server versions add native compare-and-delete commands; check what yours supports before reaching for a script. Renewal needs the same compare, with PEXPIRE in place of DEL.

-- release.lua: delete the key only if it still holds my token
-- KEYS[1] = lock key, ARGV[1] = my token
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

# Runnable in redis-cli against a single Redis server.
127.0.0.1:6379> SET lock:invoice:72 "7f3a9c1e5b" NX PX 30000
OK                      # acquired: key was absent, expires in 30 000 ms
127.0.0.1:6379> SET lock:invoice:72 "b81d02aa47" NX PX 30000
(nil)                   # refused: someone holds it

# Release only if the stored token is mine (script from above, passed inline):
127.0.0.1:6379> EVAL "if redis.call('GET',KEYS[1])==ARGV[1] then return redis.call('DEL',KEYS[1]) else return 0 end" 1 lock:invoice:72 7f3a9c1e5b
(integer) 1             # 1 = deleted, 0 = not the owner (or already expired)

# Renew only if still the owner (same shape, PEXPIRE instead of DEL):
127.0.0.1:6379> EVAL "if redis.call('GET',KEYS[1])==ARGV[1] then return redis.call('PEXPIRE',KEYS[1],ARGV[2]) else return 0 end" 1 lock:invoice:72 7f3a9c1e5b 30000

Note what this protects. The compare-and-delete protects release ownership: you cannot remove somebody else's lock. It does not protect the business write, and a random token says nothing about which owner is newer. Both of those need the fencing mechanism below.

When acquisition fails

A failed acquisition has two reasonable meanings. For scheduled or periodic work (the billing example), losing means someone else is doing it, so skip and wait for the next tick. For work that must happen (rebuilding a cache entry someone is reading), you wait and retry. Retrying callers should back off with jitter, otherwise a hot key turns into a polling storm against the lock service; see Retry & Backoff. Set a maximum wait too. A caller that waits for a lock indefinitely has turned a coordination fault into an outage.

Here is the failure that justifies everything after it. Worker A acquires a 30-second lease. It stalls for 45 seconds, because of a long garbage-collection pause, a hypervisor freeze or live migration, a laptop-style suspend, heavy swapping, or a network partition. The lease expires at 30 s. Worker B acquires the lock at 31 s and updates the invoice. At 45 s Worker A resumes and writes, using authority it no longer has.

A: acquire lock with owner token A (lease 30 s)
A: pause for 45 s   (GC pause, scheduler stall, network partition)
lock: lease expires at 30 s
B: acquire lock with owner token B
B: update invoice
A: resumes and updates invoice using stale authority

The lock service behaved correctly. It never granted two valid leases at the same time. Worker A did not necessarily know its lease had expired, because nothing in its own execution flow changed: the pause is invisible from the inside, and the next instruction runs as if no time had elapsed. “I still think I own it” is a belief about the past, not a write authorisation.

Checking the lease before every write is not enough

The tempting patch is for the worker to test lease.isValid() right before each write. It narrows the window but cannot close it. The process can be paused between the check and the write. Even if it is not, a request that left the host before expiry can sit in a network buffer or a retransmission queue and arrive after another worker has been granted the lock. This is a time-of-check to time-of-use race across machines. The only party that can resolve it is the one that receives the write, which is the next section.

A fencing token is a monotonically increasing generation number issued on every successful acquisition by a suitable coordination authority. A receives token 41; after A's lease expires, B receives 42. The protected resource accepts a write only if the token is not older than the newest it has accepted, and rejects writes from older generations once a newer one has taken effect. In the earlier timeline, A's late write carries 41, the resource has already seen 42, and the write is refused.

Where the comparison happens decides everything

Sending X-Fence: 42 in an HTTP header protects nothing if the downstream service ignores it. The database or the resource adapter has to evaluate the token atomically with the write, in the same statement or transaction. A random UUID identifies an owner but is not a fencing token, because two UUIDs cannot be ordered. The token source must also never reset or reissue a number after failover; a counter that restarts from 1 silently breaks the guarantee.

The idea is not exotic. Chubby, Google's lock service, handed out sequencers that downstream servers could verify. Kafka's transactional producers use an epoch to fence zombie instances that were replaced after a pause. Both are the same pattern: a generation number issued by the authority and enforced by the party that takes the write.

Where the token comes from

• A counter in a durable, transactional store. The lease table below keeps the counter on the same row as the lease and raises it inside the acquisition statement. It never restarts as long as the row is not deleted.
• ZooKeeper. The czxid of the lock znode (the ensemble-wide transaction id that created it), or the sequence number of the ephemeral sequential node under a parent that persists. Both increase.
• etcd. The revision of the lock key, such as its create_revision. Revisions are a cluster-wide, strictly increasing integer.
• Redis. A counter via INCR is monotonic on one server, but a failover to a replica that missed the latest increments can hand out a number that was already used. Treat it as a source only if your replication and failover story rules that out.

-- PostgreSQL 9.5+ schema plus parameterized SQL templates for an application client.
-- Replace :owner and :fence with bound parameters (e.g. $1, $2 in node-postgres).
-- These statements are NOT directly executable in psql as written.
-- The lease row's counter doubles as the fencing token.
CREATE TABLE lock_leases (
  lock_key   text        PRIMARY KEY,
  owner      text        NOT NULL,            -- for humans and dashboards
  fence      bigint      NOT NULL,            -- raised on every successful acquisition
  expires_at timestamptz NOT NULL
);

-- Acquire. Returns one row (the new fence) if you got the lease, zero rows if someone else holds it.
INSERT INTO lock_leases (lock_key, owner, fence, expires_at)
VALUES ('invoice:72', :owner, 1, now() + interval '30 seconds')
ON CONFLICT (lock_key) DO UPDATE
   SET owner      = EXCLUDED.owner,
       fence      = lock_leases.fence + 1,
       expires_at = EXCLUDED.expires_at
 WHERE lock_leases.expires_at <= now()
RETURNING fence;

-- Renew. Zero rows means the lease expired or someone else took it: stop.
UPDATE lock_leases
   SET expires_at = now() + interval '30 seconds'
 WHERE lock_key = 'invoice:72' AND fence = :fence AND expires_at > now()
RETURNING expires_at;

-- Release. Expire the lease but keep the row, so the counter never restarts at 1.
UPDATE lock_leases
   SET expires_at = now()
 WHERE lock_key = 'invoice:72' AND fence = :fence;

Guarding the write

For a single row, the guarded write is an ordinary conditional UPDATE that carries the token:

-- Parameterized PostgreSQL template; :fence and :job_id must be bound by a client.
-- The fence column lives on the row being protected.
ALTER TABLE invoice_jobs ADD COLUMN last_fence bigint NOT NULL DEFAULT 0;

-- Every write by a lock holder carries its fence. <= (not <) lets the same holder
-- make several writes with one token; a newer token still locks older ones out.
UPDATE invoice_jobs
   SET status = 'SENT', last_fence = :fence
 WHERE job_id = :job_id
   AND last_fence <= :fence
   AND status = 'PENDING';
-- 1 row  -> you were the newest authority and the transition was legal.
-- 0 rows -> you are stale, or the work is already done. Re-read before deciding which.

This rejects older tokens once a newer one has been recorded, and nothing more. Two details matter in practice. First, a strict last_fence < :fence refuses the holder's own second write with the same token, so any holder that writes more than once needs <=. Second, it is not a state machine: repeating a transition such as SENT → SENT, or work that already left the building, needs its own handling. The status = 'PENDING' precondition above is the minimum.

Fencing does not cancel work in flight

A late write by A that reaches the resource before any write by B has recorded token 42 will succeed, because the resource has nothing newer to compare against. Fencing protects an ordering boundary, not a magic cancellation of requests already on the wire. If that matters, the new owner should raise the fence as its first action, in the same transaction as the read that tells it what state it inherits.

-- Parameterized PostgreSQL template; :fence and :job_id are client-bound values.
-- Illustrative: do this first thing after acquiring, in the same transaction as the read.
BEGIN;
UPDATE invoice_jobs
   SET last_fence = :fence
 WHERE job_id = :job_id AND last_fence <= :fence
RETURNING status;                 -- 0 rows: a newer holder exists, abandon the attempt
-- ... decide from the status you just read, do the guarded writes ...
COMMIT;

Try the two orderings below. With claiming off, a stale write that arrives first is accepted, and B then proceeds from state it read before that write. With claiming on, the stale write is refused wherever it lands.

What fencing cannot protect

The resource has to understand tokens. A payment API, an email provider, or a physical actuator usually does not, and a token in your database says nothing about what the other system has already done. For those you need a stable idempotency key sent on every attempt, durable local state, and reconciliation against the provider; see Idempotency. Object stores and similar services that offer conditional writes (ETag or generation preconditions) can serve as a compare-and-set for the same purpose. Check what your service actually offers.

Put together, a worker follows the lease with a local check to stop early on known expiry, and treats the resource as the final authority:

# Illustrative Python; acquire_lease, plan, db and GUARDED_UPDATE stand for your own code.
def process_invoice(invoice_id: str) -> str:
    lease = acquire_lease(f"invoice:{invoice_id}", ttl_ms=30_000)   # returns None if someone holds it
    if lease is None:
        return "skipped"
    try:
        for step in plan(invoice_id):
            lease.ensure_valid()                    # local monotonic-clock check, stops early on known expiry
            changed = db.execute(GUARDED_UPDATE, fence=lease.fence, **step.params)
            if changed == 0:                        # the resource, not the lease, has the final say
                raise LeaseLost(invoice_id)
        return "done"
    except LeaseLost:
        return "abandoned"                          # another holder owns the job now; do not retry blindly
    finally:
        lease.release()                             # compare-and-delete or fence-matched UPDATE

A lease stops a crashed worker from holding a lock forever, but the TTL has to be chosen against two numbers that pull in opposite directions: how long the protected work can legitimately take, and how long you can tolerate waiting for recovery after a crash. A 30-second lease around a two-minute export is unsafe unless the holder renews it. A one-hour lease around a ten-second task means a crash blocks the next run for close to an hour. There is no universally right number, and rules of thumb such as “twice the P99” ignore the second constraint.

Renewal must verify ownership

A renewal is a conditional extension: extend only if the stored owner or fence is still mine, as in the Redis script and the lease-table UPDATE above. An unconditional PEXPIRE can resurrect a lock that already belongs to someone else. If a renewal fails, the worker should stop initiating new protected side effects and assume it lost authority. It should treat anything already in flight as possibly still going to happen, since cancelling a client call does not recall a request that has been sent.

Renewal interval and lease length are one decision. Renewing at a third of the lease tolerates one failed renewal; renewing at the lease length tolerates none, and a perfectly healthy worker can lose its lock to a single slow round trip. Explore the trade-off below. The worker freezes just as a renewal falls due, which is the worst moment.

A shorter lease recovers faster after a crash and also expires faster under a harmless stall, so it raises renewal traffic and the chance that a slow-but-alive worker loses ownership. The figure reports the worst-case freeze a lease absorbs (lease minus renewal interval) and, if the freeze is longer, the stale window during which only the resource's fence stands between you and a conflicting write.

Clock assumptions

A lease is a time-based promise, so ask whose clock measures it. A consensus-backed service measures it on the service side. The client must measure the same interval locally, using a monotonic clock started before the acquisition request was sent, minus a safety margin for drift and round-trip time. Wall clocks can jump (NTP steps, VM resume), so never compare timestamps taken on different machines, and never decide expiry from a client timestamp stored in the lock value. The lease table above avoids the problem by letting the database's now() be the only clock.

When the lock service itself is unavailable, you must decide in advance. Failing open (proceed without a lock) risks duplicates; failing closed (do not start work) risks starvation. Neither is right in general. Work whose duplicate is harmless can fail open. Work protected by an invariant at the resource can often fail open too, because the invariant does the real job. Work that has nothing else protecting it should fail closed.

A lease is only as safe as the service that grants it, and these systems fail in different ways. Choose by what a double-grant costs you, and add fencing regardless of the choice.

Redis with SET NX PX

One Redis server gives you a fast, simple lock with exactly the semantics you have seen: atomic acquire, expiry by TTL, compare-and-delete by script. The weakness is availability and replication. A primary with an asynchronous replica acknowledges the write before copying it. If the primary dies in between and the replica is promoted, the new primary has never heard of the lock, and another client acquires it while the first still holds it. WAIT can make acknowledgement depend on replicas receiving the write, which narrows the window, but it does not turn Redis into a consensus system.

Redlock

Redlock tries to remove the single-point problem. The client acquires the lock on a majority of, say, five independent Redis masters within a time budget, and treats the remaining validity as the lease time minus the time spent acquiring and a drift allowance. The debate is a fair one. Critics, most prominently Martin Kleppmann, argue that it rests on timing assumptions (bounded delays, bounded pauses, bounded clock drift) and that it offers no fencing token, so a paused client can still act after expiry. Its author argues that the assumptions are acceptable and that the client can check remaining validity after acquiring. Both sides agree that a pause after the final check cannot be excluded. A practical reading: Redlock is reasonable for locks whose failure only costs efficiency, and it should not be the sole protection for something that corrupts durable state. If you need that, put a fence at the resource or use a consensus-backed service plus a fence.

ZooKeeper

The standard recipe creates an ephemeral, sequential znode under a lock path. The client with the lowest sequence number holds the lock, and every other client watches only the node immediately before its own, not the lowest, so releasing wakes one waiter instead of the whole herd. Ownership is tied to a session: if the session expires (no heartbeat within the session timeout), the server deletes the ephemeral nodes. The ordering comes from consensus, so double-grants from failover are not the main risk. The remaining one is the familiar one: a paused client only learns its session expired after it resumes and reconnects, and may act in between. The znode's czxid is the natural fencing token.

etcd

etcd grants a lease with a TTL, which the client keeps alive with periodic keep-alive requests. A lock is a key attached to that lease, and the owner is the key with the lowest creation revision under a lock prefix. If the keep-alives stop, the lease expires and its keys are deleted. Revisions form one strictly increasing sequence across the cluster, so the lock key's revision is the natural fencing token. As with ZooKeeper, consensus makes the grant itself trustworthy; it does not make a frozen client aware that time passed.

Relational databases

If the work happens in the same database as the lock, the lock and the data share one transaction and one failure domain, and the stale-owner problem mostly disappears. When the connection drops, the transaction aborts and its row locks are released together with its writes. This is why the next section comes first in practice.

If all competing work updates the same relational database, the simplest correct solution is often a transaction, a unique constraint, or an atomic compare-and-set update. Inserting a row under a unique (customer_id, billing_period) key makes the database guarantee a single invoice, whichever worker gets there first or how many are paused.

-- PostgreSQL schema is executable; the INSERT uses client-bound :customer_id and :period.
-- The database itself guarantees one invoice per customer and period.
CREATE TABLE invoices (
  invoice_id       bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  customer_id      bigint NOT NULL,
  billing_period   date   NOT NULL,
  status           text   NOT NULL DEFAULT 'DRAFT',
  UNIQUE (customer_id, billing_period)
);

-- All three workers run this. Exactly one gets a row back; the others get zero and skip.
INSERT INTO invoices (customer_id, billing_period)
VALUES (:customer_id, :period)
ON CONFLICT (customer_id, billing_period) DO NOTHING
RETURNING invoice_id;

The oversold-inventory problem has the same shape. The race comes from reading and then writing, so fold the two into one statement. When a row has to be read, thought about, and written back, a version column gives optimistic concurrency without holding anything:

-- Parameterized PostgreSQL templates; bind :sku, :new_balance, :id and :version_i_read.
-- The race from "read stock, then write stock" disappears
-- because the check and the decrement are one atomic statement.
UPDATE inventory
   SET quantity = quantity - 1
 WHERE sku = :sku AND quantity > 0;
-- rowcount 1 -> reserved; rowcount 0 -> sold out. No lock service involved.

-- Optimistic concurrency on a versioned row:
UPDATE accounts
   SET balance = :new_balance, version = version + 1
 WHERE account_id = :id AND version = :version_i_read;

Row locks and advisory locks

SELECT ... FOR UPDATE serialises changes to existing rows for the length of a transaction, subject to the isolation level and how much the transaction covers. SKIP LOCKED and NOWAIT avoid waiting. PostgreSQL advisory locks coordinate cooperating sessions around an application-chosen key. The transaction-level variants release automatically at commit or rollback, and session-level locks live until explicitly released or the session ends. Their lifetime semantics differ from a leased external lock: there is no TTL, and a session-level lock can outlive the work it was meant to cover. Behind a transaction-pooling proxy such as PgBouncer, session-level advisory locks are unsafe, because the session that took the lock may not be the one that later tries to release it.

-- PostgreSQL template: bind :job_id in the first SELECT before executing through a client.
-- Row lock for the length of one transaction (PostgreSQL).
BEGIN;
SELECT status FROM invoice_jobs WHERE job_id = :job_id FOR UPDATE;      -- others wait
-- ...read, decide, write... all inside the same transaction...
COMMIT;                                                                  -- lock released

-- Do not wait: take a different row or give up
SELECT job_id FROM invoice_jobs WHERE status = 'PENDING'
 ORDER BY job_id LIMIT 1 FOR UPDATE SKIP LOCKED;

-- Advisory lock tied to a transaction (PostgreSQL). Released on COMMIT/ROLLBACK or if the connection drops.
BEGIN;
SELECT pg_try_advisory_xact_lock(hashtext('invoice:72'));   -- true = you hold it, false = someone else does
-- ...work that only touches this database...
COMMIT;

None of these makes an external payment call happen exactly once. They coordinate the database; they do not reach out and constrain a provider.

Other ways to avoid the lock

• Idempotent operations. If repeating the operation is harmless because it carries a stable key, you no longer need to prevent concurrent execution. Duplicates become cheap instead of forbidden.
• Serialise per key through a queue. Routing every message for a key to one ordered consumer serialises that key's work with no lock. See Queues & Workers.
• Static ownership. If a given piece of work should always go to the same node, assign it once instead of locking on every operation. Partition ownership is often decided by hashing; see Consistent Hashing.

Use an external distributed lock when coordination really does span independent workers or resources, and the protected operation cannot be expressed as a single atomic transition in the authoritative store. When there is an invariant that can live at the data store, put it there.

It helps to ask what happens if the lock occasionally fails. If the answer is “some work is done twice and wasted”, you have an efficiency lock. A cron job on five nodes where a rare duplicate run only recomputes a report is a good example. A single Redis instance with a TTL is a perfectly reasonable tool there. If the answer is “money moves twice” or “durable state is corrupted”, you have a correctness lock, and the lock alone is not allowed to be the defence. It needs a fence or an invariant at the resource.

• Scheduled and cron work. Without coordination every instance fires. A lease with a skip on failure turns five attempts into one. Add an idempotent write so a rare double run is harmless.
• Cache refresh. One node rebuilds an expensive entry while the others wait or serve slightly stale data. A duplicate rebuild costs CPU, so an efficiency lock is enough.
• Resource allocation. Assigning a unique slot, a partition or a device to one consumer. If a wrong double assignment is dangerous, the assignment table should have a unique constraint on the resource.
• Leader election. A lock held for a long time and kept alive by renewal. The lock's fencing token is the leader's epoch, and followers or downstream resources should reject commands from an older epoch. A deposed leader that has not noticed yet is exactly the stale owner from earlier.
• Payment deduplication. A duplicate webhook is common, so a lock can cut attempts, but the real guard is an idempotency key and a unique constraint; see the payout example below.

A payout scheduler obtains a Redis lock and calls a payment provider. The provider completes the payout, but the scheduler times out before receiving the response, and its lease expires. A second worker acquires the lock, which is a legitimate non-overlapping grant, and retries the payout. The provider has no reason to suspect a repeat.

Both workers held the lock, at different times, and two payouts occurred. The external side effect is not coupled to the lock at all, so no amount of tuning the lock fixes this. The safe design relies on three other things:

1. A stable idempotency key. Derive it once from the payout id and send the same key on every attempt, so the provider can return the original result for a repeat. Providers expire these keys after some retention window, so check theirs.
2. Durable local state. Record intent (SUBMITTING) before the call, so a retry can see that an attempt may already be in flight.
3. Reconciliation. A job that finds rows stuck in SUBMITTING and asks the provider what happened, by key, instead of guessing.

-- Illustrative. Durable intent first, a stable idempotency key on every attempt, guarded transitions.
CREATE TABLE payouts (
  payout_id     text   PRIMARY KEY,
  amount_cents  bigint NOT NULL,
  state         text   NOT NULL CHECK (state IN ('PENDING','SUBMITTING','SUBMITTED','FAILED')),
  provider_key  text   NOT NULL UNIQUE,      -- derived once from payout_id, sent on every attempt
  provider_ref  text,
  last_fence    bigint NOT NULL DEFAULT 0
);

-- 1. Before calling the provider (retry after a crash re-enters SUBMITTING with a newer fence).
UPDATE payouts SET state = 'SUBMITTING', last_fence = :fence
 WHERE payout_id = :id AND state IN ('PENDING','SUBMITTING') AND last_fence <= :fence;

-- 2. Call the provider with Idempotency-Key = provider_key. A repeat returns the original result.

-- 3. Record what the provider answered.
UPDATE payouts SET state = 'SUBMITTED', provider_ref = :ref, last_fence = :fence
 WHERE payout_id = :id AND state = 'SUBMITTING' AND last_fence <= :fence;

-- 4. Reconciliation job: rows stuck in SUBMITTING are looked up at the provider by provider_key.

The lock is still useful here. It cuts the number of concurrent attempts so the provider and your database see less redundant traffic. It just cannot be the thing that prevents the double payout.

What to monitor

Track lock contention and wait time, renewal failures, critical-section duration against the lease length, stale-write rejections from the fence (each is a stale owner you caught), and owner churn for a given key. A rising contention rate can mean the critical section is too long, the key is hot, or a dependency is stuck. Raising the TTL without knowing which one it is lengthens recovery and can make an incident worse. A non-zero count of fence rejections is not a bug to silence; it is the mechanism doing its job and telling you how often pauses really happen.

Use a distributed lock for bounded coordination when competing actors can honour the same lock authority, and when stale actions are either rejected by the resource or made harmless by idempotency. Avoid making it the only defence against duplicate payments, overselling inventory, or corrupting durable state. Walk the options in this order:

1. Can a unique constraint, a conditional UPDATE or a transaction on one database express it? Then use that and skip the lock.
2. Is the operation naturally idempotent, or can it be made so with a key? Then duplicates are survivable, and a lock is only an optimisation.
3. Does coordination genuinely cross workers or systems? Use a lease, size it against work and recovery time, and renew with an ownership check.
4. Does a stale actor cause real damage? Enforce a fencing token at the resource, and pick a token source that cannot repeat after failover.
5. Is the side effect external and fence-ignorant? Add idempotency keys, durable state and reconciliation, and treat the lock as a traffic reducer only.

Skip the lock entirely for a single-instance application, where a local mutex or a transaction is enough, and for read-only paths, which never need exclusive access.

The guard that makes repeats harmless is covered in Idempotency, and the same lease-and-generation scheme applied to job rows is in Job Processing. When a business process spans several services and cannot share one lock or transaction, the Saga Pattern coordinates the steps and their compensations instead. For publishing a change and an event without a lock across the two, see the Transactional Outbox, and for deciding which node owns which key without a lock per operation, Consistent Hashing.