Context
A financial platform serving 7M+ users. Account creation depended on an external financial provider, and the backend retried failed provider calls automatically.
Incident
Around 10 a.m., more than 100 customers failed account creation at nearly the same time. Each of them had been retried more than 10 times, and every retry failed with the same provider response: “Customer Already Exists.”
Investigation
- The Command Center caught the error spike in Grafana. My part was the logs.
- In AWS CloudWatch, I rebuilt the sequence of provider calls for affected customers, starting from the first request rather than the latest error.
- The first request had failed differently: the provider created the customer but did not complete the account, because the due date was invalid.
- That made “Customer Already Exists” a secondary error: the provider was correctly refusing to create the same customer twice.
- Teammates who knew the provider's business rules confirmed which due dates it rejected.
Root cause
- Partial creation: on the first call, the provider created the customer and stopped before creating the account.
- An undocumented provider rule: the due date could not be later than day 28. Our flow did not enforce it.
- Retries masked the cause: each retry repeated customer creation against a customer that now existed, so the provider answered “Customer Already Exists.” The real reason appeared only in the first response, under ten or more identical retry failures.
My contribution
- Investigated the CloudWatch logs and reconstructed the request flow for the affected customers.
- Identified the invalid due date as the original failure, and “Customer Already Exists” as a side effect of the retries.
Fix and recovery
Done as a team:
- Validation in both frontend and backend, so due dates after day 28 no longer reach the provider.
- Automated tests for the day-28 rule.
- A recovery script for stuck customers that skipped customer creation, since the provider had already done that step.
Result
- About 100 customers recovered.
- Resolved in about 4 hours.
- This error stopped generating support tickets after the fix.
Engineering takeaway
Automatic retries are safe only when the downstream call is all-or-nothing. When a provider can fail halfway, a retry is no longer the same request: it runs against new state and gets a new error. In a chain of calls, find the first failure for one affected request before trusting the error everyone is looking at, and make recovery resume from the step that already succeeded.