Production Engineering

Race Condition

Why two perfectly valid concurrent operations can produce an invalid result — and why the same code can work a thousand times and then fail once, just because of timing.

Mangesh MandlikLast updated: 2 November 2026

01 · The Problem

What Is a Race Condition?

A race condition happens when the correctness of a program depends on the timing or ordering of concurrent operations that access shared state.

Take a shared counter starting at 100. Two requests, A and B, each want to increment it. Run one after another, and the answer is obviously 102.

Request A

→

Write 101

Request B

→

Write 102

Final = 102 ✓

Safe sequential execution: each request sees the other's result before it starts.

But run them concurrently, and a naive implementation can produce a different answer:

Request A

Request B

Write 101

Final = 101 ✗ (expected 102)

Race: both requests read the old value before either one writes — one update overwrites the other.

02 · The Mechanism

Why Does It Happen?

counter++ looks like one operation in source code. It usually isn't.

Read → Modify → Write

Incrementing a shared value conceptually involves three separate steps: read the current value, compute the new value in memory, and write it back. Each step is a distinct moment in time.

Steps From Different Requests Can Interleave

If two requests each run this three-step sequence at roughly the same time, their individual steps can interleave in any order the scheduler happens to pick — that's what makes the outcome depend on timing.

A: LOCK

→

Read·Modify·Write

→

UNLOCK

then

B: WAIT

→

LOCK → RMW → UNLOCK

Final = 102 ✓

Protected: a lock forces the three steps of one request to finish before the other request's steps can begin.

Whichever mechanism eventually fixes it, the root cause is always the same: an operation that looks atomic in code is actually several steps, and shared, mutable state is reachable by more than one of those step-sequences at once.

03 · Animated Demonstration

Watch It Happen

Press play to watch the safe sequential run, then the same code racing and losing an update, then two real-world flavors of the same bug, then four different fixes.

Production Engineering

Race Condition

⤢ Expand for synchronized narration timeline

04 · Impact

Why Is It Dangerous?

A race condition doesn't crash your service. It quietly produces the wrong answer — and only sometimes.

🔢Lost Updates

One write silently overwrites another. Counters drift, totals stop adding up, and nothing in the logs says why.

🎲Intermittent Failures

The exact same code can run correctly thousands of times and fail once, only when a particular interleaving occurs.

👻Vanishes Under Debugging

Attaching a debugger or adding logging changes the timing enough that the bad interleaving stops happening — right when you're looking for it.

📈Worse Under Load

More concurrent requests means more opportunities for a bad interleaving, so the bug often shows up only in production, under real traffic.

💰Data / Business Impact

When the shared state is an account balance or inventory count, a lost update becomes an overdrawn account or an oversold item — not just a wrong number.

🤫No Error Signal

There's typically no exception and no crash — just quietly incorrect data that surfaces later, often in a reconciliation report.

05 · Real-World Examples

Real-World Examples

Counters are the textbook case. The same bug shows up anywhere concurrent operations read, compute, and write shared state.

💳

Account Balance

Balance = ₹100. Request A withdraws ₹80, Request B withdraws ₹50 — both arriving at almost the same instant. Both read ₹100 and both compute independently. A writes ₹20; B writes ₹50 on top of it. The system just approved ₹130 of withdrawals against ₹100 that ever existed — with no error raised.

📦

Inventory

Inventory = 1. Two buyers click 'Buy' at the same time. Both see 1 unit available, both proceed, and both get a confirmation. The warehouse only has one unit to ship — the item was oversold by the exact same read-then-write race.

06 · Mitigation

How Do Engineers Fix It?

There is no single universally correct fix. The right mechanism depends on the shared state and how often conflicts actually happen.

Mutex / Lock

Only one operation can execute the critical section — the read-modify-write sequence — at a time. In one process that is often a mutex; everyone else waits.

How it works

Request A acquires the lock, runs its full read-modify-write sequence, and releases it. Request B, which arrived while the lock was held, waits — then acquires it and runs safely afterward.

Trade-off

Simple mental model and strong correctness guarantees, but introduces blocking and contention. Overused or nested incorrectly, locks can also lead to deadlocks and reduced concurrency, and a process-local mutex does not protect shared state across multiple server instances.

Atomic Operations

For simple operations like counters, an atomic increment or compare-and-swap performs the read-modify-write as one indivisible step at the hardware, runtime, or database level.

How it works

The platform guarantees no other operation can observe or interleave with a partially-completed update — for example, an atomic increment or a single SQL update like `SET value = value + 1` leaves no read-then-write gap to race through.

Trade-off

Extremely fast and avoids blocking entirely, but only applies to the narrow set of operations the platform can make atomic. Doesn't help with multi-step business logic.

Database Transactions / Row Locks

For persistent shared data, the database can protect a critical update — but usually via an atomic SQL statement, explicit row locking, or optimistic conflict detection, not merely by wrapping code in a transaction.

How it works

Transaction A might lock the row with `SELECT ... FOR UPDATE`, or issue a single atomic `UPDATE` that changes the value in one step. Transaction B, trying to touch the same row, either waits or hits a write conflict depending on the isolation and mechanism in use.

Trade-off

Strong consistency for persisted state without custom application logic, but long-held row locks increase database contention and the isolation level and locking strategy need to be chosen deliberately.

Optimistic Concurrency

Instead of locking up front, tag each row with a version. Write conditionally, and detect conflicts instead of preventing them.

How it works

Read value + version (e.g. version = 10). Update 'where version = 10'. If another transaction already changed the row, the update simply fails — the caller re-reads the fresh value and retries.

Trade-off

Avoids blocking almost entirely and scales well when conflicts are rare, but requires explicit conflict-handling and retry logic in the application.

Also be clear about scope: a mutex inside one application process protects only that process. If multiple instances can touch the same state, coordination has to happen through the shared system itself — for example database row/version checks or a distributed lock.

And if the production bug is really duplicate submits or safe retries, idempotency helps make repeated requests safe. It does not by itself prevent two true concurrent read-modify-write operations from overwriting each other.

Technique	Best For	Benefit	Trade-off
Lock / Mutex	Critical sections	Simple mutual exclusion	Contention / blocking
Atomic operation	Simple shared state	Fast, no blocking	Limited applicability
Atomic SQL / row lock	Persistent shared data	Strong consistency	DB-side contention
Optimistic concurrency	Infrequent conflicts	Less blocking	Retries / conflict handling
Reduce shared state	Architecture / design	Less synchronization needed	May require redesign

07 · Diagnosis

Diagnosing Race Conditions

Race conditions rarely announce themselves. Look for a pattern of symptoms, not a single error.

🔢Numbers That Don't Add Up

Incorrect counters, drifted totals, lost updates, or inconsistent state with no corresponding error in the logs.

👻Bugs That Disappear Under Debugging

Attaching a debugger or slowing execution down changes the timing — and the failure often stops reproducing.

📈More Frequent Under Load

Higher concurrency means more chances for a bad interleaving, so the failure rate climbs with traffic.

🧪Only Reproduces Under Stress

The failure shows up reliably only when the same code path is run repeatedly and concurrently, under real contention.

Useful techniques: stress testing and repeated concurrent execution of the same code path, concurrency testing tools built to force specific interleavings, thread/stack dumps where applicable, and careful logging around state transitions so a bad interleaving leaves a trace when it does occur.

08 · Takeaway

Engineering Takeaway

Key Takeaways

A race condition is not simply "two requests running at once." It occurs when concurrent operations access shared state in a way where the final result depends on their timing or ordering.
Concurrency itself is not the problem — it's necessary for performance and scale. The danger is unsynchronized access to shared mutable state.
"counter++" is really Read → Modify → Write. Any of those steps can interleave with another request's steps on the same shared state.
The bug is timing-dependent and intermittent — the same code can work a thousand times and fail once, and it often gets worse under production load.
The key engineering question is: "what shared state can be accessed concurrently, and what guarantees do we need around it?"
Then choose deliberately: an atomic operation, a lock, an appropriately isolated transaction or row/version check, optimistic concurrency — or, often best, redesign to reduce shared mutable state. Don't reach for locks everywhere by default.