Idempotency: why a retried payment gets charged twice

A customer clicks “Pay”, the connection lags, they click again, and they’re charged twice. Or your payment provider retries a notification that did arrive but whose response got lost, and your system records the payment twice. In both cases the fault is on your side, and in both the fix is the same: make sure that repeating an operation doesn’t carry it out twice.

When it shows up

You don’t need high volume for this. It shows up with ten customers, through these four routes:

• Double click. The button isn’t disabled, or it’s disabled only in the browser of whoever tested it.
• Browser or network retry. A request that times out may have been executed anyway. The client has no way of knowing.
• Provider retry. Every serious payment provider retries a notification if it doesn’t get a proper response within a few seconds. Stripe keeps retrying for up to three days.
• Your own queue retrying. A job that fails after charging and before recording that it charged goes back into the queue and charges again.

The last two are the ones that surprise people most, because the system works perfectly for months until the day latency spikes.

Why the obvious fix doesn’t work

Almost everyone’s first attempt is to check before acting:

Is there already a charge for this order? If not, charge it.

That fixes nothing, and it’s worth understanding why, because the same reason sinks plenty of other fixes.

Time passes between the check and the write. Very little, but it passes. If two requests arrive at once, and a double click sends them at once by definition, both check before either has written, both find nothing, and both charge. The window is milliseconds wide and gets hit more often than intuition suggests.

The second attempt is usually a lock in the application code. It works as long as there’s a single instance. It stops working the day there are two, which tends to be the day the product starts doing well.

The pattern: the idempotency key

The operation stops being identified by what it does and starts being identified by who asked for it, and which request it was.

The client generates a unique ID for the attempt itself (which isn’t the same as the order) and sends it with the request. The server stores that ID in the same transaction that performs the operation, with a uniqueness constraint in the database. If a second request arrives with the same key, the constraint fails, and instead of running again the server returns the result of the first.

Three details make it actually work:

1. Uniqueness is enforced by the database. It’s the one point in the system where two simultaneous requests can’t both win. Code alone can’t guarantee that.
2. The key and the effect are stored together, in one transaction. Store them separately and the window you wanted to close is back.
3. The response is stored too. The second attempt should get exactly what the first one got. For the caller, the operation succeeded once, and that’s all it needs to know.

Serious payment providers already support this on their side: Stripe accepts an idempotency key so that a retried request doesn’t perform the same operation twice. That covers your call to them. It doesn’t cover your own records, which is usually where the real problem is: the charge happened once and you recorded it twice.

Incoming notifications are the same problem in reverse

When the payment provider tells you a payment has completed, your endpoint has to assume three uncomfortable things:

• The same notification can arrive more than once. That’s the normal case.
• They can arrive out of order. The “subscription canceled” notification can arrive before the “subscription updated” one.
• One can arrive for an event you already knew about through another route, because your application had already processed the synchronous response.

It’s solved the same way: every provider event comes with its own ID. Store it with a uniqueness constraint, process it inside the transaction, and if it was already there, return a success response and do nothing else. Responding with an error to a duplicate notification is an expensive mistake: the provider will keep retrying for days.

And one check that often gets forgotten: verify the notification’s signature. A payment webhook endpoint without signature verification is an endpoint where anyone can declare they’ve paid.

What it costs to apply

No pattern is free, and this one comes with a few small costs:

• One more table and one more constraint, which need cleaning up. Keys are retained for a reasonable time, from hours to days depending on the provider, and then purged.
• A little more latency. Writing the key sits on the operation’s critical path.
• Discipline on the client. The key is generated by the caller, and it has to be the same on the retry. A new key per attempt protects against nothing, and it’s the most common implementation mistake.

On that last one: if your client generates a random ID on every send, you’ve built the whole mechanism for nothing. The key identifies the intent, so every send of that intent carries the same key.

What still fails

With everything done right, two cases are worth keeping in mind:

An operation that touches two systems. Charging with the provider and recording it in your database are two writes in two places, and no transaction spans both. You can make each one idempotent, but there’s still a moment when one has happened and the other hasn’t. It’s handled by reconciling against the provider periodically: asking which charges really exist and comparing them with yours. It’s boring, but it’s the only thing that works.

Refunds. Almost everyone protects the charge and leaves the refund unprotected. The gap is discovered the day someone refunds the same amount twice.

When it isn’t worth it

If your operation has no side effects (a query, a search), it’s idempotent by nature and there’s nothing to do.

And if the effect is reversible and cheap, daily reconciliation sometimes makes more sense than up-front protection. Sending a notification email twice is annoying; charging twice is an incident with your customer and their bank. Put the effort where a mistake costs money.

What’s not up for debate: any operation that moves money, issues a legal document or notifies a third party gets protected from day one. Adding it later means auditing what happened during the unprotected period, and that audit costs more than doing it right in the first place.

---

Vecinly charges recurring subscriptions in production, with failed payments, plan changes and incoming notifications from the payment provider. You can see it in its case study. If you have a prototype that already takes payments and you’re not sure it’s protected, custom SaaS starts by reviewing exactly that.