Avoiding Double Refunds: How We Solved a Race Condition in Our Payout System

When two processes — the webhook response and the status-check job — handled failed payouts simultaneously, our users sometimes got refunded twice. This post explains how our payout system encountered a concurrency issue and how we fixed it using optimistic locking.

---

💥 The Problem

At first glance, our payout flow looked simple:

1. A user requests a payout.
2. We send the request to our acquirer.
3. The acquirer sends a webhook with the final status (success or failed).
4. Separately, we run a scheduled status-check job to catch missed webhooks.
5. If a payout fails → we refund the amount to the user’s wallet.

But occasionally, both the webhook and the job would detect a failure at almost the same moment — and both triggered refunds. The result? The user’s wallet balance increased twice 💸💸.

---

This concurrency race condition wasn’t frequent — but in financial systems, even one duplicate refund is unacceptable.

---

⚙️ The Solution: Optimistic Locking

Instead of introducing complex distributed locks, we implemented a lightweight optimistic locking mechanism at the wallet level.

How It Works

Optimistic locking assumes that conflicts are rare but possible. It works like this:

• Each wallet row has a version number (an integer column).
• When a process wants to update the wallet balance, it checks the version number it last read.
• The update query succeeds only if the version hasn’t changed.
• If another process already modified the wallet, the current update fails gracefully — triggering a retry or log instead of a duplicate refund.

Example schema:

ALTER TABLE wallets ADD COLUMN version INT DEFAULT 0 NOT NULL;

Example update query:

UPDATE wallets
SET balance = balance + 100, version = version + 1
WHERE id = 123 AND version = 5;

If no row is affected (because the version has changed), the application knows someone else already updated it — avoiding double refunds.

---

---

🔑 Benefits

Implementing optimistic locking brought immediate benefits:

• 🧩 Prevents double refunds or incorrect balances even under high concurrency.
• ⚡ No heavy database locks — safe for distributed systems and jobs.
• 🔁 Easy to implement for any entity where state accuracy is critical.
• 🤝 Works perfectly with redundancy, like webhook + scheduled job architecture.

---

🧪 Testing the System

Before deploying, we stress-tested the new logic to ensure it handled real-world concurrency.

Test Scenarios:

• Simulate concurrent payout failures from both webhook and job processes using load-testing tools like JMeter or Artillery.
• Only one refund update should succeed; others should detect a version conflict.
• Check wallet balances after each test — they must remain consistent.

Expected Outcome:

• ✅ One refund succeeds.
• ⚠️ The second detects a version conflict and exits.
• 💰 Wallet balance remains correct.

---

🧭 Lessons Learned

1. Race conditions don’t always appear in development — but they always exist in distributed systems.
2. Optimistic locking provides a clean, scalable safeguard without slowing down transactions.
3. Monitoring and observability are just as important — logs must clearly show conflicts and retries.
4. Small design improvements like this can save massive financial losses and improve user trust.

---

Final Thoughts

In payment systems, concurrency control is just as critical as correctness. By adding a version-based optimistic lock to our wallet updates, we eliminated duplicate refunds — without adding latency or operational complexity.

Simple fix. Huge impact.

— Protize Engineering Team