Context
I worked on a high-volume financial platform serving more than 7 million active users. Account creation depended on a downstream financial provider.
Problem
A subset of customers could not complete account creation. The provider first rejected some requests because of an invalid payment due-date rule. Later retries returned “Customer Already Exists”, hiding the original error and sending the team down the wrong path.
Investigation
- I traced the production request flow in CloudWatch logs and request tracing.
- I compared successful and failed requests to see where their behavior diverged.
- I worked with the external provider to understand why retries returned a different response from the first request.
Root cause
The frontend allowed unsupported due-date values to reach the downstream integration. The provider accepted due dates only up to day 28. After the first failure, retries produced a different, misleading error.
My contribution
- Traced the request flow and identified the original failure.
- Isolated the external integration behavior.
- Helped create a recovery path for affected accounts.
- Helped stop invalid due dates from reaching the integration.
Outcome
Invalid due dates stopped reaching the provider, and affected accounts had a recovery path.
What I learned
Retries can hide the original failure. If the downstream error changes on the second attempt, the first response is still the one that matters.