When Metrics Backfire: The Cobra Effect in the age of AI
2026-11-05T17:58:37+00:00Author: Tomasz Byrski Your metrics dashboard might be suggesting you [...]
Author: Tomasz Byrski Your metrics dashboard might be suggesting you [...]
Authors: Dylan Sharkey Erica Padilla Saurabh Shertukde Walter Litwinczyk Harshad Sheth Introduction Modern fintech systems must support diverse financial primitives that began life as separate products: cross-border remittance settlement, payroll direct deposit, card issuing and acquiring, and revolving credit. Each carries its own consistency model, latency profile, and regulatory treatment. Delivering a unified account experience across them, without a tightly coupled backend monolith or a fragmented customer journey, is a distributed systems problem rather than a product packaging exercise. The Remitly Global Card combines four of these behind a single program: customers get paid into it, spend from it, send money home, and borrow against it when eligible. A wallet balance is authoritative cash held now, a loan is an obligation amortized over months, a card authorization must resolve within a sub-second window under a strict SLA, and a remittance settles asynchronously through a partner rail. To power these constructs, we engineered an orchestration architecture that unifies identity verification, card issuing lifecycles, real-time credit underwriting, risk assessment, and multi-rail payment routing. The core engineering challenge was composing heterogeneous domain services into a fault-tolerant backend without leaking domain complexity to client applications. What follows traces how we got there, from where the initial version fell short to the boundaries that hold the systems together. Architectural Evolution & The Legacy Silos Platforms like this rarely start unified. Each capability ships as its own product with its own integration path, and the seams overlap once one account has to present all of them. Our initial product was assembled from legacy point-to-point integrations, leading to structural problems: Incompatible ledgers and circular dependencies: The wallet maintained a double-entry cash ledger of what a customer held while the lending engine kept an interest-accruing schedule of what the same customer owed. Neither domain could answer the other's half and ended up relying on it for data it did not own. Consequently, logic followed the data: credit-line functions into the wallet, membership logic into lending. These cycles meant any change required coordinated releases across teams that did not own it. Client orchestration: Applications made redundant RPCs across distinct identity endpoints and assembled the results themselves. State restoration on retry was fragile to an extent where abandoned/stuck flows could not be resumed. Synchronous coupling: Direct calls between independent microservices produced cascading failures during partner outages, propagating latency with an unpredictable blast radius. A single card service owning the combined experience was considered and rejected. It would have shipped quickly, but every change after would have required negotiation with one team holding authority over a state it was not accountable for. Technical Orchestration & The Core Invariants We established specific constraints to prevent expensive failure modes before drawing any service boundaries: Strict balance contexts: Liquid cash and revolving credit remain isolated in distinct micro-ledgers. If a domain goes down, the system degrades safely instead of projecting inaccurate balances to the customer. Domain ownership boundaries: Identity owns verification state, underwriting owns risk decisions, and payments owns movement. Deterministic audit trails: State transitions write append-only records capturing customer intent, ensuring disputed transactions can be reconstructed. Asynchronous resilience: Network timeouts resolve through explicit state machines. If a customer taps submit twice during a timeout, an idempotency key ensures they are charged only once. Edge abstraction: Core logic relies on standardized internal protobuf schemas, while vendor-specific integrations are isolated inside edge adapters. One Experience, Clear Ownership (The Layered Architecture) The layers are where those invariants are enforced. What matters is not the number of tiers but the prohibition attached to each, and the table isolates the four where ownership was drawn firmly: Layer Owns Does not own Experience gateway Customer-facing GraphQL schema and aggregation Authoritative balances or transaction execution Onboarding orchestrator Versioned onboarding flow, checkpoints, resumable progress Screen rendering, identity truth, credit decisions, payments Card product layer Cross-domain product rules and composition The internal lifecycle of any domain Domain services Authoritative state and domain lifecycle The combined customer journey The Experience Layer at the Edge At the edge network, a high-throughput GraphQL federation gateway handles field-level query resolution across domain subgraphs in parallel. DataLoader batching, response caching and circuit breakers serve complex frontend state at low latency while isolating clients from partial downstream failures. The layer is deliberately read-oriented. A balance resolves directly from the service that owns it, and card state comes from the card domain. If a domain goes down, the cost is one unresolved field the client gracefully handles instead of rendering a fabricated zero balance. Directly beneath it, the card product layer orchestrates multi-step sagas and owns the rules that exist only because the product composes several domains: which actions are available in context, whether a funding source is valid for a send, how a composite status reads. Logic reusable within one domain remains there, which keeps the layer thin. Onboarding vs. Money Movement: Two Coordination Planes The layers determine where the state lives, but not how work moves through them over time. Products pairing a long-running enrollment journey with real-time money movement may be tempted to couple them and apply identical design patterns. The two have different convergence requirements and one mechanism serves neither well. We decoupled them into distinct coordination planes, operating on different clocks and resolving to different owners. The Unified Onboarding System Onboarding is a long-running journey with pauses: identity checks, requests for additional information, asynchronous decisions, customers returning days later. Rather than rebuild that per product, our orchestration engine models a journey as checkpoints, with guards deciding when one can be entered and actions doing the work inside it. Each journey runs as a durable Temporal workflow, so it can pause on a pending input and leverage event-driven state hydration to resume days later without holding a thread. The engine is flow-agnostic. It advances checkpoints, evaluates guards and runs actions by name, and knows nothing about KYC, underwriting or card issuance; those live in a product layer that depends on the engine and never the other way round. Adding a step is a product change, which is why one orchestrator now runs onboarding for lending, wallet and membership. Definitions are versioned and immutable, so a journey in flight keeps the one it started on. The backend decides which requirement comes next, the frontend decides its presentation, and the journey changes without a client release. What the orchestrator is prohibited from doing enforces the boundary: it advances a journey and never executes a transfer, just as the experience gateway never originates a balance or a credit decision. These negative definitions are as important as the arrows in the diagram. Borrowing and Payments Stay Separate The money-movement plane is the other shape, a lifecycle that must converge rather than pause. Wherever credit sits beside a stored balance (e.g. wallet), there is pressure to merge the two into one spendable number. That pressure should be addressed at the experience layer and resisted at the ledger. Borrowing tracks what a customer has drawn against a line, what remains owed and when it falls due. Payments execute the movement of money without needing to know what it is for, which is what keeps it general: a loan repayment, a membership charge and an ordinary transfer all travel the same path. Drawing credit does move money. It produces two records rather than one: borrowing registers the draw, and payments funds the transfer, not the wallet, orchestrated so that both commit or neither does. Repayment reverses the pair, and borrowing marks the obligation satisfied only once payments are confirmed. Collapsing the two into a single ledger would not simplify the system. It would make it wrong: an obligation and a spendable balance are distinct instruments with distinct accounting and regulatory treatment. The experience layer is where the two come together as a single product offering. What a customer reads as a single sense of what they can spend is composed from both sources, and never persisted as one number. Failure Is a State, Not an Exception Financial systems fail ambiguously. A request can time out after a partner has already accepted it, and an event can arrive twice or out of order. Treated as generic errors, these outcomes produce duplicate movements and unexplainable states. The simplest instance is a duplicate submission: a customer taps send twice, or a client retries a request it never received a response to. The system processes two requests but produces one movement because the idempotency key is scoped to customer intent rather than the request itself. Money-moving operations must be built on idempotent boundaries, durable identifiers, explicit state transitions, and reconciliation. The goal is not a system that never fails, but one where every failure has a name, an owner, and a clear path back to a known state. Why the Shape Holds The practical test of these boundaries is the change they can absorb without requiring a rewrite. A new payment rail is simply an edge adapter requiring no change to underwriting. A new borrowing product receives its own lifecycle and needs no new wallet ledger. A new market relies on edge mappings rather than a fork of the core system. Because no tier holds another tier's state, engineering teams are able to release against published contracts rather than each other. Key Architectural Takeaways The card brings four money moments together: getting paid, spending, sending money home and borrowing. Each still has exactly one owner underneath it, so the customer sees one plan while every fact in it traces back to the domain accountable for it. Unify customer intent, not systems of record. A coherent product can be composed above independently owned domains, and multi-domain workflows can be orchestrated without cross-domain database joins. An aggregation layer should assemble relationships, not duplicate truth. Federated subgraphs make distributed information usable at low latency while authority remains with the source. An orchestrator coordinates; a domain decides. The onboarding orchestrator advances a journey without owning identity or financial state, and the same constraint governs cross-domain composition. Recovery is part of the product. Explicit pending states, reconciliation and operational tooling are customer-facing features even when customers never see the machinery. Keep the center stable and the edges replaceable. Scale comes from common contracts and modular adapters rather than uniform partner behavior. Conclusion & Looking Ahead Engineering the Remitly Global Card platform required solving distributed coordination, real-time ledgering, and fault-tolerance challenges across distinct domain primitives. By establishing strict domain authority, clear invariants, and distinct coordination planes, we created a scalable foundation capable of handling millions of global customers with strict reliability and regulatory compliance. None of these are just card patterns. Any system comprising credit, stored value and cross-border settlement behind one account confronts the same questions about ownership, convergence and failure, and the answers transfer even when the primitives differ. This architecture serves as our core template for scalable financial engineering. In upcoming posts, we will dive deeper into individual subsystems: GraphQL DataLoader optimization patterns, Temporal workflows, Saga pattern implementation in Steelix, real-time two-phase credit ledgering, and edge adapter integration patterns, the loan management system in borrow and more.
