Payments are a distributed system: the 28-second void
In 2014 we looked after a system that took payments from a payment provider and passed them on to the systems that held customer accounts.
The rules we were playing by
- The provider wanted an answer to each payment message within 28 seconds.
- Answer late and it sent a void message straight away.
- It kept sending voids until something answered.
- End to end, our side had to receive the payment, credit the account on the downstream system, and reply. If that took longer than 28 seconds, the void was already on its way.
What went wrong
Some accounts ended up with extra credit, and the provider seemed to be sending duplicate voids even when we answered within the second.
The investigation found two things:
- Incoming payment and void messages were never checked against ones we'd already seen. Every duplicate void was processed, so each one became another negative top-up sent downstream.
- Failed top-ups to the downstream systems were retried on a schedule. Voids weren't. Where a void hit a communication error, it was simply dropped.
So some voids landed several times, some not at all, and account balances stopped matching the payments.
The fix
- De-duplicate on the way in. For any duplicate message, send back the reply we gave the original. The provider gets the same answer every time, and nothing is processed twice.
- Put voids in the same retry queue as top-ups.
Testing it was its own job: the provider couldn't send duplicates on demand, so the test system had to be slowed down past the 28 seconds to make the provider send them.
What I took from it
A payment that crosses three systems is a distributed system, whatever the spec calls it.
Every message can arrive twice, late or not at all. Handlers need to be idempotent, and every kind of message needs a retry, not just the happy path one.
It also needs the release discipline in deploy rules for a small agency: this went to test and was proved by the provider before it was ready for live.