All engineering cases
CASE 01 / Production debugging · Fintech

Finding the real failure behind a misleading financial API error.

On a financial platform with 7M+ active users, customers were blocked in onboarding by a provider error that did not match the original failure. I traced the request path and isolated the first error.

Context

I worked on a high-volume financial platform serving more than 7 million active users. Account creation depended on a downstream financial provider.

Problem

A subset of customers could not complete account creation. The provider first rejected some requests because of an invalid payment due-date rule. Later retries returned “Customer Already Exists”, hiding the original error and sending the team down the wrong path.

Investigation

  1. I traced the production request flow in CloudWatch logs and request tracing.
  2. I compared successful and failed requests to see where their behavior diverged.
  3. I worked with the external provider to understand why retries returned a different response from the first request.

Root cause

The frontend allowed unsupported due-date values to reach the downstream integration. The provider accepted due dates only up to day 28. After the first failure, retries produced a different, misleading error.

My contribution

  • Traced the request flow and identified the original failure.
  • Isolated the external integration behavior.
  • Helped create a recovery path for affected accounts.
  • Helped stop invalid due dates from reaching the integration.
THE RESULT

Outcome

Invalid due dates stopped reaching the provider, and affected accounts had a recovery path.

What I learned

Retries can hide the original failure. If the downstream error changes on the second attempt, the first response is still the one that matters.
NEXT CASE / 02Investigating database timeouts under e-commerce traffic
Discuss my experience