18 November 2026

Race Condition Testing for Payment Workflows

Race Condition Testing for Payment Workflows
9 min. readBy Borg

Race Condition Testing for Payment Workflows

Race condition testing for payment workflows means running the same sensitive operation concurrently in an authorised sandbox, then checking durable state (ledger rows, balances, provider objects) to see whether a business rule broke. The HTTP responses are the least reliable part of the evidence.

A race condition is a bug that only exists in the gaps between steps. A refund handler reads the order, checks the refundable amount, then writes the refund. Two requests that arrive close together can both pass the check before either has written anything. Each request, viewed alone, did nothing wrong. Together they refunded twice.

This guide is for backend and platform engineers, security engineers and CTOs who own payment, credit, refund or subscription flows. It covers where these flows break, how to set up a safe test, how to test idempotency boundaries instead of just idempotency keys, and how to turn a result into a regression test. It stays bounded and defensive: no payloads, no high-volume recipes, no real money.

What a race condition is in a payment workflow

A race condition is a defect where the outcome depends on the timing or ordering of concurrent operations. A payment workflow is any flow where that timing decides who gets charged, credited or refunded. The textbook definition covers threads and shared memory. In payments the shared resource is usually a database row, a balance, a coupon, or an object at a payment provider.

The common shape is check-then-act. The code checks a condition (balance sufficient, coupon unused, refund not yet issued), then acts on it (debit, redeem, refund). If nothing makes the check and the act one atomic step, concurrent requests can all see the same pre-action state.

So what is the difference between a deadlock and a race condition? A race condition produces a wrong result because operations interleave unsafely. A deadlock produces no result because two operations each hold something the other needs and both wait forever. Fixes can interact too: add coarse locking to cure a race and you can introduce a deadlock if lock ordering is inconsistent. Test for both.

The OWASP Web Security Testing Guide covers this under business logic testing for payment functionality, and frames it the same way. Test authorised systems, focus on check-then-act operations and on actions that should succeed only once (OWASP WSTG).

Where payment workflows break under concurrency

Payment workflows break wherever a rule of the form "this should happen at most once" or "this total must not exceed that limit" is enforced in application code instead of by the database or the provider. A hypothetical list of the usual suspects:

  • Refunds: the cumulative refunded amount must never exceed the captured amount. Two concurrent partial refunds can each look valid in isolation.
  • Balance and credit debits: read balance, compare, write new balance. Without a conditional update or row lock, the second writer overwrites the first.
  • Single-use coupons and credits: redemption is a check-then-act on a flag.
  • Subscription changes: upgrade, downgrade and cancel arriving together can leave proration or entitlements inconsistent with what was billed.
  • Payout and transfer requests: the same pending balance can be claimed twice if reservation is not atomic.
  • Webhook handlers: the provider can deliver the same event more than once, or out of order, while your own request path is also updating the same order. For signature and delivery checks specifically, see webhook security testing beyond signature checks. This guide only cares about the concurrency between the two paths.

The double-spend and double-charge cases are the same bug class with different stakes. If the invariant lives in a code comment instead of a constraint, assume it can break under concurrency until a test shows otherwise.

Most of these are authorisation-adjacent business logic problems, which is why scanners tend to miss them. Nothing is malformed. Every individual request is legitimate.

Set up an authorised sandbox test before you send anything

Set up the test environment and the rules of engagement first. The dangerous part of concurrency testing is not the technique. It is running it somewhere it can cause harm.

A reasonable checklist:

  1. Written authorisation. Name the systems, the time window, and who to call if something misbehaves. Concurrency tests can leave data in odd states.
  2. Provider test mode only. Use your payment provider's sandbox or test keys. No real cards, no real balances, no real payouts.
  3. Seeded, disposable data. Create dedicated test accounts, orders and balances. Record starting state before every run so you can compare afterwards.
  4. Minimal concurrency. A small number of near-simultaneous requests is enough to expose a missing atomic check. If two or three overlapping requests cannot break the invariant, a flood is unlikely to teach you more. A flood also risks rate limits and noisy side effects.
  5. Define the invariant before the test. Write it down: "total refunds for an order never exceed the captured amount", "a coupon is redeemed once", "a balance never goes negative". A test without a stated invariant produces anecdotes, not findings.
  6. Isolate from production side effects. Suppress outbound emails, fulfilment and accounting exports for test accounts.

Language-level race detectors find data races inside a single process. These catch in-memory races. They do not catch the cross-request, cross-service races that matter most in payments, which live in the database and at the provider. For those, write an integration test that fires a small number of concurrent calls at a real sandbox stack and asserts on stored state.

Test idempotency boundaries, not just idempotency keys

Testing idempotency means checking what actually happens at the edges of the key's protection, not only that a repeated key returns the same answer. Idempotency keys are a good control, and they have boundaries.

Stripe's documentation describes the behaviour for API v1. Once endpoint execution starts, the first response (including a 500) is saved and returned for retries with the same key, and reusing a key with different parameters produces an error. If validation fails, or a concurrent request conflict prevents execution from starting, no result is saved and the request can be retried. Keys can be pruned after at least 24 hours, and API v2 behaves differently from v1 (Stripe docs). Check the current documentation for the version you use.

Those details translate into concrete boundary questions for your own system:

  • Key scope. Is the key unique per logical operation, or does your client reuse one key across different refunds or orders? A reused key with changed parameters should fail loudly. Confirm it does.
  • Where the key is honoured. If you pass a key to the provider, but your own database write happens before or after that call with no guard of its own, the provider is idempotent and your ledger is not.
  • Key generation on retry. A client that mints a fresh key on every retry defeats the mechanism. A hypothetical example: a mobile app retries after a timeout with a new random key, and the first request had in fact succeeded.
  • The conflict path. Concurrent requests with the same key can be rejected instead of deduplicated. Test whether your caller treats that rejection as "failed" and retries with a new key.
  • Expiry. After the retention window a reused key may be treated as a new request. Decide how your system reconciles uncertain outcomes before retrying late.
  • Keys on non-payment side effects. The provider's key protects the provider call. It does not protect your entitlement grant, your email, or your internal ledger entry.

The goal is a table of boundaries with an expected and observed result for each. A single pass or fail tells you little.

Verify from durable state, not HTTP responses

Verify outcomes from durable state. An HTTP response only tells you what one request believed happened, and under concurrency that belief can be wrong in both directions.

Two requests can both return 200 while only one change persisted. Or one returns an error after the write committed, so the client retries and the operation runs twice. The OWASP guidance makes the same point: send multiple identical requests close together, then verify the final state and not just the response codes. Look for duplicate processing, multiple redemptions, negative balances or allocations beyond the limit (OWASP WSTG).

After each run, read these directly:

  • Ledger rows. Count entries for the operation. One logical refund should be one row, or one balanced pair of rows.
  • Balances. Compare the final balance with starting balance plus the sum of ledger entries. A mismatch is a finding even if every response looked fine.
  • Provider objects. List the charges, refunds or transfers in the sandbox for the test account and compare them with your own records. A divergence in either direction matters.
  • Derived state. Entitlements, credits, invoice status and any counters.
  • Event trail. Webhook deliveries and your handler's processing log, so you can see ordering.

Write the assertion as an invariant check against those sources, not as "expect status 200". A useful habit: run a reconciliation query after every test and treat any nonzero discrepancy as a failure.

This is also the standard for a finding. A claim that a flow is racy should be backed by durable evidence and reproduction steps, which is the idea behind verified security findings.

Turn the result into a regression test and a finding

A confirmed race should produce two artefacts: a regression test that fails before the fix and passes after, and a finding that an engineer can act on without a meeting.

The finding should contain the invariant that broke, the starting state, the minimal concurrent sequence, the durable-state evidence (row counts, balance delta, provider object IDs from the sandbox) and the affected code path. Keep it free of anything that reads as a weaponised recipe. The purpose is to let the owner reproduce it in their own sandbox.

Fixes tend to fall into a few families, and the right one depends on where the invariant should live:

  • Atomic database operations. A conditional update (UPDATE ... WHERE balance >= amount) or a transaction with row locking, so check and act cannot be separated.
  • Unique constraints. One ledger row per idempotency key or per operation ID, enforced by the database, so a duplicate insert fails.
  • Idempotency controls end to end. Honour the key in your own write path, not only in the provider call.
  • Atomic token invalidation. For single-use items, consume and validate in one step.

The regression test should assert on durable state, use a small fixed number of concurrent calls, and run in CI against the sandbox stack. Expect some flakiness risk: concurrency tests can pass by luck, so run the case repeatedly and keep it narrowly scoped. For how to keep fixes from quietly regressing, see security regression testing for authorisation fixes. The same approach applies here.

Test the invariants you cannot afford to break

Pick the three payment invariants that would hurt most if they failed (refund totals, balance floors, single-use credits), write each as a durable-state assertion, and run a small authorised concurrency test against your sandbox this sprint. If you would rather have verified findings with reproduction steps than build the harness yourself, Borg tests payment and business-logic flows against the running application, with source-code context, and retests after fixes ship. Book a Demo at borghq.io.

Frequently asked questions

What is the definition of a race condition?
A race condition is a defect where the result depends on the timing or ordering of concurrent operations. In code it often means a check-then-act sequence that is not atomic, so two operations both pass the check before either acts. In payments, that can mean duplicate refunds, double debits or a single-use credit redeemed twice.
What is the difference between a deadlock and a race condition?
A race condition gives a wrong result because concurrent operations interleave unsafely. A deadlock gives no result because operations wait on each other indefinitely. They can be linked: adding locks to fix a race can create a deadlock if lock ordering is inconsistent. Concurrency tests should look for both wrong outcomes and hangs.
How do you test race conditions in payment workflows?
Test in an authorised sandbox with provider test keys and disposable data. State the invariant first, send a small number of near-simultaneous identical requests, then verify from ledger rows, balances and provider objects rather than HTTP responses. Turn any confirmed break into a regression test that asserts on durable state.
Do idempotency keys prevent double charges?
They help, within limits. A key protects the call it is attached to, and Stripe's documentation notes details such as saved first responses, errors on changed parameters, retryable conflict cases and key pruning after at least 24 hours. They do not protect your own ledger, entitlements or other side effects unless your write path honours them too.
How do you fix a race condition in a payment flow?
Move the invariant into something atomic. Common fixes are conditional database updates, transactions with locking, unique constraints on operation or idempotency IDs, and atomic invalidation of single-use tokens. Then add a regression test that checks durable state, so the fix does not quietly erode in later changes.

Borg on
Social Media

Borg specializes in securing complex, high-risk digital systems.