Race Conditions: What They Are and How to Handle Them

I once spent three days chasing a bug where users were getting charged twice for the same order. The code looked correct every time I read it. Tests passed. The bug only appeared under load, when multiple requests hit the same endpoint within milliseconds of each other. It was a race condition, and I had no idea what that meant until I was knee-deep in one.

A race condition happens when the behavior of a program depends on the timing of events, specifically when two or more operations read and write shared state without coordinating with each other. The "race" is between those concurrent operations, and whoever gets there first changes what the other one sees.

The maddening part: race conditions don't show up reliably. They're timing-dependent, which means they can disappear under a debugger (which slows things down), pass in tests (which run fast in isolation), and then detonate in production under real load.

Why race conditions happen

The root cause is almost always this pattern: read a value, make a decision based on it, then write a new value back. If two operations do this at the same time on the same piece of shared state, both reads happen before either write, and both operations make their decision based on stale data.

Here's a concrete example. You have a function that gives a user a discount, but only if they haven't already claimed one:

async function claimDiscount(userId: string) {
  const user = await db.users.findById(userId)
  if (user.discountClaimed) return { error: 'Already claimed' }

  await db.users.update(userId, { discountClaimed: true })
  await applyDiscount(userId)
}

If two requests for the same user arrive at the same time, both will read discountClaimed: false, both will pass the check, and both will apply the discount. The check is useless without coordination.

This is the check-then-act pattern. It's the source of most race conditions in application code.

Locking: mutual exclusion

The oldest solution is a lock. Before reading shared state, you acquire a lock. Other operations that need the same lock will wait. When you're done, you release it.

At the database level, this looks like a row-level lock:

BEGIN;
SELECT * FROM users WHERE id = $1 FOR UPDATE;
-- now no other transaction can modify this row until we commit
UPDATE users SET discount_claimed = true WHERE id = $1;
COMMIT;

FOR UPDATE locks the row for the duration of the transaction. Any concurrent query trying to do the same thing will block until the first transaction commits or rolls back.

Locking works, but it comes with a cost. If a high volume of requests target the same row, they queue up. Throughput drops. If locks are held for too long, you can see cascading slowdowns. And if two transactions each hold a lock the other needs, you get a deadlock, where both are waiting forever.

Race conditions vs deadlocks

These two terms get confused, but they are different problems. A race condition is two operations reading stale state and both proceeding when only one should: the result is incorrect data. A deadlock is two operations each waiting for the other to release a lock, so neither can proceed: the result is a hang.

Deadlocks typically happen when locks are acquired in different orders. Transaction A locks row 1 then tries to lock row 2. Transaction B locked row 2 and is trying to lock row 1. Neither can proceed. The fix is to always acquire locks in the same order across all code paths, or to set a lock timeout and retry on failure.

PostgreSQL and MySQL detect deadlocks automatically and abort one of the transactions with an error. But detection takes time (usually hundreds of milliseconds) and the aborted transaction needs to be retried at the application level. The database handles the detection; you handle the recovery.

SELECT FOR UPDATE SKIP LOCKED

FOR UPDATE makes waiting requests queue up behind the lock. SKIP LOCKED changes that behavior: instead of waiting, a query skips any row that is already locked and takes the next available one. This makes it useful for building worker queues directly inside a database, without a separate message broker:

BEGIN;

SELECT * FROM jobs
WHERE status = 'pending'
ORDER BY created_at
LIMIT 1
FOR UPDATE SKIP LOCKED;

-- process the job here

UPDATE jobs SET status = 'done' WHERE id = $1;
COMMIT;

Multiple workers can run this query simultaneously. Each grabs a different row because they skip locked ones. No Redis, no Kafka, no external queue. Just your existing database.

The limitation: it only works when any available item is acceptable. If every worker needs a specific row, they all contend on that same row and SKIP LOCKED does not help. It is a queue pattern, not a general locking strategy.

Optimistic locking: assume no conflict, handle it if it occurs

Optimistic locking takes a different approach. Instead of blocking concurrent access, it allows it and then detects conflicts at write time.

The typical implementation adds a version column to the row. When you read the row, you note the version. When you write, you include the version in your WHERE clause:

UPDATE users
SET discount_claimed = true, version = version + 1
WHERE id = $1 AND version = $2;

If another transaction modified the row first, its version will have changed, and your update will affect zero rows. You detect that and retry or return an error to the user.

Optimistic locking is great when conflicts are rare. It avoids holding locks, so throughput stays high. But under heavy contention on the same row, you can end up with many failed retries, which is its own kind of problem.

Atomic operations

For simple counters and flags, you can skip locking entirely by using an atomic operation: a single read-modify-write that the database or underlying system executes as one indivisible step.

Most databases have this:

UPDATE inventory SET quantity = quantity - 1 WHERE id = $1 AND quantity > 0;

This decrement is atomic. You're not reading the value, computing a new one in your application, and writing it back. You're letting the database do the whole thing in a single operation. Concurrent requests each get their decrement applied correctly, because the database serializes them internally.

In application code, languages provide atomic primitives too. In Go, sync/atomic. In JavaScript running on a single thread, the event loop gives you implicit atomicity within a single synchronous chunk of code. But the moment you await something, you've yielded control and another operation can run before you resume.

Queues and serialization

Sometimes the right answer is to not handle concurrent writes at all. Instead, funnel the requests through a queue and process them one at a time.

For the discount-claiming example: instead of multiple server instances all hitting the database directly, you push a claim_discount job to a queue. A single worker processes jobs one at a time, in order. Concurrency is eliminated at the point where it matters.

This is a meaningful architectural decision, not just a trick. You're trading throughput (parallelism) for correctness (serialization). For operations where correctness matters more than raw speed, it's often the right call.

Redis is commonly used for this with its SETNX command (set if not exists), which lets you implement a distributed lock:

const acquired = await redis.set(`lock:discount:${userId}`, '1', 'NX', 'EX', 10)
if (!acquired) return { error: 'Try again shortly' }

try {
  await claimDiscount(userId)
} finally {
  await redis.del(`lock:discount:${userId}`)
}

The lock is scoped to the user, so different users don't block each other. The expiry (EX 10) ensures the lock is released even if the server crashes.

Fencing tokens: handling the zombie lock problem

The Redis distributed lock has a subtle failure mode that the expiry does not fully solve. Consider this sequence:

1. Worker A acquires the lock with a 10-second expiry
2. Worker A pauses (a long garbage collection cycle, a slow network call, a VM getting preempted)
3. The lock expires
4. Worker B acquires the lock and starts processing
5. Worker A resumes, unaware its lock expired, and runs concurrently with B

Setting a longer expiry delays the problem but does not eliminate it. The lock holder can always pause longer than the expiry.

Fencing tokens close this gap. Every time a lock is acquired, the lock server returns a monotonically increasing token number. That token is passed along with every write to the protected resource:

Worker A acquires lock → gets token 42
(A pauses, lock expires)
Worker B acquires lock → gets token 43

Worker B writes with token 43 → accepted, resource stores 43 as current token
Worker A resumes, tries to write with token 42 → rejected, 42 < 43

The resource (the database, object store, or any downstream system) tracks the highest token it has seen and rejects any write carrying a lower or equal token. A zombie process that woke up after its lock expired carries a stale token and cannot cause damage.

In practice, you store the current token in the database and check it as part of the conditional write:

UPDATE resources
SET data = $1, fence_token = $2
WHERE id = $3 AND fence_token < $2;

If the row already has a higher token, the update affects zero rows. The caller detects that and knows its lock was superseded.

Fencing tokens require the resource being protected to cooperate: it must check the token on every write. That is often an application-level convention rather than something the lock server enforces automatically. If you use Redis for distributed locking on a high-stakes operation, fencing tokens are the missing piece that makes it actually safe.

Isolation levels: what your database guarantees by default

When multiple transactions run concurrently, the database has to decide how much each transaction can see of the others' in-progress work. That decision is the isolation level, and it directly determines which race conditions are possible without additional locks.

There are four standard levels, each protecting against a different class of problem:

Read Uncommitted: A transaction can read data written by another transaction that has not yet committed. You can read data that gets rolled back, a "dirty read". Almost never used in practice.

Read Committed: You only see committed data. Dirty reads are prevented. But if another transaction commits a change between two of your reads of the same row, you see different values, a "non-repeatable read". This is the default in PostgreSQL and most OLTP databases.

Repeatable Read: Once you read a row, you always get the same value for that row within your transaction, even if someone else commits a change. But you can still see new rows inserted by others, a "phantom read". This is the default in MySQL/InnoDB.

Serializable: Full isolation. Transactions behave as if they ran one at a time in some serial order. No dirty reads, no non-repeatable reads, no phantom reads. The cost is throughput: more conflicts, more transaction aborts, more retries under contention.

The practical implication: at Read Committed (the PostgreSQL default), another transaction can slip between your read and your write in a check-then-act pattern. At Serializable, it cannot, but you pay in throughput. Most applications leave isolation at the default and compensate with explicit locks or atomic operations where correctness matters most.

You can set the level per transaction:

BEGIN ISOLATION LEVEL REPEATABLE READ;
-- your queries
COMMIT;

Knowing your database's default is the starting point. If you have never checked, look it up: it shapes every concurrent operation your application runs.

Idempotency as a safety net

No matter which technique you use, it's worth making the operation itself idempotent: calling it multiple times produces the same result as calling it once.

For payments, this usually means generating an idempotency key on the client side and storing it with the transaction:

INSERT INTO charges (id, user_id, amount, idempotency_key)
VALUES ($1, $2, $3, $4)
ON CONFLICT (idempotency_key) DO NOTHING;

If a duplicate request comes in, the insert silently does nothing. The client gets back a success response pointing at the original charge. No double billing, even if the same request was sent twice due to a network retry.

How ON CONFLICT works

ON CONFLICT is a PostgreSQL feature that handles what happens when an INSERT would violate a unique constraint. The column you name in ON CONFLICT (...) must have a UNIQUE or PRIMARY KEY constraint on it. When a duplicate is detected, instead of throwing an error, Postgres takes the action you specify.

DO NOTHING silently drops the duplicate. DO UPDATE lets you upsert by updating specific columns on the existing row:

ON CONFLICT (idempotency_key) DO UPDATE
  SET status = EXCLUDED.status,
      updated_at = NOW();

EXCLUDED refers to the row that would have been inserted, so you can selectively merge values from it.

The important property for concurrency: the check and the insert are a single atomic operation. There is no gap between "does this key exist?" and "insert if not" where a second request could sneak in. A separate SELECT followed by an INSERT would have that gap and could still double-insert under concurrent load.

How the frontend should generate the key

The client generates a UUID v4 before sending the request. Every modern browser exposes this natively:

const idempotencyKey = crypto.randomUUID()

The key is attached to the request as a header and, critically, kept alive across retries:

async function charge(userId: string, amount: number) {
  const idempotencyKey = crypto.randomUUID()

  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      return await fetch('/api/charges', {
        method: 'POST',
        headers: { 'Idempotency-Key': idempotencyKey },
        body: JSON.stringify({ userId, amount }),
      })
    } catch {
      if (attempt === 2) throw
    }
  }
}

The rule is: same user action, same key. New user action, new key. If the user clicks "Pay", the network drops, and the client retries automatically, all three attempts carry the same key and the server deduplicates them. If the user sees an error and clicks "Pay" again themselves, that is a new action and should get a new key, so a fresh charge can go through.

The most common mistake is generating the key inside the retry loop. That makes every retry look like a brand new request to the server, defeating the whole point.

For operations that span multiple steps (a multi-page checkout, for example), store the key in sessionStorage tied to the checkout session. Clear it once you receive a confirmed success response. A local variable works fine for single-function retries like the example above.

Idempotency doesn't eliminate the race condition, but it limits the damage. Combined with a lock or atomic operation, it gives you defense in depth.

SQL vs NoSQL: does the database change anything?

The race condition itself is the same problem regardless of the database. What differs is how much the database helps you close the window.

SQL databases give you strong concurrency guarantees by default. ACID transactions, row-level locks, ON CONFLICT, and atomic updates are all built in. You declare what you need and the engine handles it.

NoSQL databases trade some of those guarantees for scale. The protections are still achievable, but you reach for them more explicitly, and the defaults are less safe.

MongoDB: single-document atomicity

MongoDB guarantees atomicity at the single-document level. If your entire operation touches one document, you get it for free:

const result = await db.users.updateOne(
  { _id: userId, discountClaimed: false },
  { $set: { discountClaimed: true } }
)

if (result.matchedCount === 0) {
  return { error: 'Already claimed' }
}

The filter and the update are one indivisible operation. If two requests race, one wins and the other matches zero documents. No lock needed, no transaction needed. This is why MongoDB encourages embedding related data in one document: it keeps operations atomic without cross-document coordination.

Multi-document operations need explicit transactions

The moment your operation touches more than one collection, atomicity disappears unless you opt into transactions:

// NOT safe: two separate writes, no atomicity between them
await db.charges.insertOne({ userId, amount, idempotencyKey })
await db.users.updateOne({ _id: userId }, { $inc: { totalSpent: amount } })

A crash or concurrent request between those two lines leaves your data inconsistent. MongoDB added multi-document transactions in v4.0 for exactly this:

const session = client.startSession()

try {
  await session.withTransaction(async () => {
    await db.charges.insertOne(
      { userId, amount, idempotencyKey },
      { session }
    )
    await db.users.updateOne(
      { _id: userId },
      { $inc: { totalSpent: amount } },
      { session }
    )
  })
} finally {
  await session.endSession()
}

You must pass session to every operation inside the block. Omitting it on even one call means that operation runs outside the transaction, which is a silent correctness bug.

Idempotency in MongoDB

There is no ON CONFLICT DO NOTHING. The equivalent is a unique index plus catching the duplicate key error:

// run once in a migration or at startup
db.charges.createIndex({ idempotencyKey: 1 }, { unique: true })

// in your handler
try {
  await db.charges.insertOne({ userId, amount, idempotencyKey })
} catch (err) {
  if (err.code === 11000) {
    return db.charges.findOne({ idempotencyKey })
  }
  throw err
}

Error code 11000 is MongoDB's duplicate key error. The behavior is the same as ON CONFLICT DO NOTHING, just more verbose and with a second read to fetch the original document.

Optimistic locking in MongoDB

MongoDB has no built-in version column, but the pattern works exactly the same way manually:

const doc = await db.inventory.findOne({ _id: itemId })

const result = await db.inventory.updateOne(
  { _id: itemId, version: doc.version },
  { $set: { quantity: doc.quantity - 1 }, $inc: { version: 1 } }
)

if (result.matchedCount === 0) {
  throw new Error('Conflict, retry')
}

DynamoDB: conditional writes and limited transactions

DynamoDB has no transactions in the traditional sense, but it has two tools for concurrency.

The first is ConditionExpression: an atomic check-and-write that only applies the operation if a condition holds. This is the equivalent of SQL's UPDATE ... WHERE or MongoDB's filtered updateOne:

await dynamodb.putItem({
  TableName: 'Charges',
  Item: { userId, amount, idempotencyKey },
  ConditionExpression: 'attribute_not_exists(idempotencyKey)',
})

If an item with that idempotencyKey already exists, the write is rejected with a ConditionalCheckFailedException. You catch that and treat it as a duplicate:

try {
  await dynamodb.putItem({ ... })
} catch (err) {
  if (err.name === 'ConditionalCheckFailedException') {
    return dynamodb.getItem({ Key: { idempotencyKey } })
  }
  throw err
}

For optimistic locking, the same pattern applies: include a version attribute in the condition, increment it on write, and catch the failure if someone else got there first.

The second tool is TransactWriteItems, which lets you bundle up to 25 operations across multiple tables into a single all-or-nothing request:

await dynamodb.transactWrite({
  TransactItems: [
    {
      Put: {
        TableName: 'Charges',
        Item: { userId, amount, idempotencyKey },
        ConditionExpression: 'attribute_not_exists(idempotencyKey)',
      },
    },
    {
      Update: {
        TableName: 'Users',
        Key: { userId },
        UpdateExpression: 'ADD totalSpent :amount',
        ExpressionAttributeValues: { ':amount': amount },
      },
    },
  ],
})

The 25-item cap is a hard limit, not a soft guideline. If your operation touches more items than that, you have to redesign your data model or accept weaker guarantees. TransactWriteItems is also more expensive than a regular write and has higher latency, so it is not something you reach for by default.

How they compare

The guarantees are achievable in all of them. The difference is how much the database helps you by default, how explicit you have to be, and what you pay in latency and throughput when you opt into stronger consistency.

The SAGA pattern: distributed transactions across services

A database transaction keeps a set of writes atomic. But once your operation spans multiple services, each with its own database, there is no transaction boundary that covers all of them. You cannot BEGIN across a service boundary.

The SAGA pattern is the standard answer. Instead of one atomic transaction, you break the operation into a sequence of local transactions, each owned by one service. If a step fails, you run compensating transactions to undo the steps that already succeeded.

Consider a booking flow: reserve the flight, charge the card, confirm the seat. These touch three different services:

There are two coordination styles. In a choreography SAGA, each service emits events and the next service reacts. No central coordinator; the flow is implicit in event subscriptions. Simpler to build, harder to reason about when things go wrong.

In an orchestration SAGA, a central process manager explicitly calls each step and drives the compensating transactions on failure. More visible, easier to debug and monitor, but the orchestrator becomes a critical component that must itself be reliable.

The hard part is not the happy path. It is writing correct compensating transactions. They must be idempotent: a compensation step might be called more than once due to retries or network failures, and they must handle partial failures gracefully. This is where most SAGA implementations go wrong.

SAGAs do not give you isolation. Between the time a step completes and its compensation runs, other operations can observe intermediate state. If that is unacceptable, a SAGA is the wrong tool. Usually it is acceptable if you design the UX around it: show a "processing" state rather than instant confirmation, and communicate clearly when a reversal has occurred.

Testing race conditions

The reason race conditions survive code review and unit tests is that they only appear when two operations overlap in time. A test that runs operations sequentially will never trigger one.

The technique is to force the overlap. You pause execution at a specific point, let another operation run to a specific point, then resume both. In JavaScript, you can do this with a manually resolved promise:

let resume: () => void

async function claimDiscount(userId: string, pauseAfterRead = false) {
  const user = await db.users.findById(userId)

  if (pauseAfterRead) {
    await new Promise<void>(resolve => { resume = resolve })
  }

  if (user.discountClaimed) return { error: 'Already claimed' }
  await db.users.update(userId, { discountClaimed: true })
  await applyDiscount(userId)
}

it('prevents double discount under concurrent requests', async () => {
  const first = claimDiscount('user-1', true) // pauses after read
  const second = claimDiscount('user-1')       // runs to completion
  await second

  resume()  // first resumes with stale data
  await first

  const user = await db.users.findById('user-1')
  expect(user.discountCount).toBe(1) // fails if the race condition exists
})

If the race condition exists, both operations read discountClaimed: false, both apply the discount, and the count ends up at 2. The test fails deterministically.

This is a concurrency test harness. It is not the same as a load test. A load test fires many requests and hopes to get lucky with timing. A harness controls timing precisely, making the race reproducible every run.

Two requirements for this to work. First, you need a real database, not a mock. Mocking the database removes the actual concurrency primitives (locks, atomic operations) that prevent or allow the race, so a mock-based test proves nothing about concurrent correctness. Second, the pause must happen at the right point: after the read, before the write. That is the exact window the race lives in.

Which approach to reach for

There is no universal answer, but here is how I think about it.

Optimistic vs pessimistic locking: if conflicts are rare (most requests for the same resource do not overlap), optimistic locking keeps throughput high and avoids the overhead of acquiring locks. If conflicts are frequent or the cost of a failed retry is high (a long computation, an external API call), pessimistic locking is safer even though it serializes requests.

Where to solve it: if the operation is simple and the data lives in one database, reach for atomic operations or database-level locking first. They are simple and well understood. If conflicts are rare and retries are acceptable, optimistic locking keeps things fast. If you need coordination across services or the operation is expensive and should only run once, a distributed lock or a queue is the right level to solve it. If the operation spans multiple services, design for a SAGA from the start rather than trying to retrofit it.

The mistake I see most often is skipping the analysis and just adding a mutex everywhere, or worse, assuming the problem does not exist until it blows up in production. Race conditions are predictable if you look for the check-then-act pattern. Wherever you read shared state and make a decision, ask: what happens if another request does the same thing right now?

If the answer is "something bad," you have a race condition. Pick the tool that fits the scale and the stakes.

For more on the database side of this, my piece on ACID transactions covers exactly what guarantees your database gives you around concurrent writes. And if you want to understand what breaks when you distribute this across services, the outbox pattern article gets into the consequences.

I write about this kind of systems thinking every week in Monday BY Gazar, a newsletter for engineers pushing toward staff and principal roles. If concurrency, correctness, and distributed design are things you think about, it's worth subscribing.