This is the depth of analysis behind every project we build. We publish it as confirmation of how we actually work — the real options we weighed, the numbers that ruled them out, the architecture that survived contact with a live authorization pipeline — so our process stays as transparent to clients as the results are. This project’s version of "carving from stone" wasn’t about picking the more accurate model — no single model type could hit the latency budget and catch behaviorally novel fraud at the same time, so the real work was arranging two disagreeing signals to agree fast enough to matter.
The client ran a digital-first payments operation processing a high volume of card-not-present transactions — e-commerce and subscription charges with no chip, no PIN, no physical card to inspect. Their fraud engine was a rules stack: hand-tuned thresholds on amount, velocity, and a handful of blocklists, refined over years by whoever was on call for the last incident. It worked in the narrow sense that it blocked a lot of things — including a lot of things that weren’t fraud. A returning customer buying a laptop at 2am from a new device on a work trip looked, to a static rules engine, exactly like account takeover. The false-positive rate on legitimate high-value transactions had become a measurable source of churn, and the fraud team couldn’t fix it without also loosening the rules that were catching real fraud.
The ask was not "add a model on top of the rules." It was: cut false positives on legitimate customers, catch more of the fraud the rules engine was missing, and do it inside the same latency window checkout already budgets for a fraud decision. That last constraint shaped almost everything below — a more accurate model that adds half a second to checkout doesn’t ship, because a slow authorization is its own kind of failure.
The second constraint that mattered as much as latency: this is a regulated financial workflow. A decision to decline someone’s card has to be explainable and reconstructable — to a compliance reviewer, a disputes team, potentially a regulator — months later, not just fast in the moment.
Why deep learning wasn’t the obvious upgrade it’s often assumed to be: Grinsztajn, Oyallon, and Varoquaux’s NeurIPS 2022 benchmark paper, "Why do tree-based models still outperform deep learning on typical tabular data" (arXiv:2207.08815), ran a controlled comparison across 45 tabular datasets and concluded — in the authors’ own words — that "tree-based models remain state-of-the-art on medium-sized data (~10K samples) even without accounting for their superior speed." We’re citing that qualitative conclusion deliberately rather than a specific point-gap number — the per-dataset tables would need to be pulled directly to quote an exact accuracy delta, and we’d rather cite what we can stand behind than round a figure we didn’t personally verify. Tabular fraud data — structured, feature-engineered, moderate-dimensional — is exactly the regime the paper documents trees winning in, not the regime (images, text, very large datasets) where deep learning’s advantage shows up instead.
Why the latency budget reinforced the same conclusion independently: NVIDIA’s published account of American Express’s "Gen X" fraud model (a compute-vendor blog, so vendor-adjacent framing, not a neutral third party) describes a production model combining gradient boosting (1,000+ trees) with an LSTM component, built to meet a stated 2-millisecond inference budget, with the GPU-accelerated version delivering roughly a 50x speedup over the prior CPU approach. We’re not claiming our latency matches Amex’s — their figure covers one component of a much larger pipeline at a scale we don’t operate at — but the pattern is the relevant data point: even one of the largest card networks in the world leans on gradient-boosted trees, not an end-to-end deep network, when milliseconds are the constraint.
Why not have the LLM do the primary scoring: an LLM API call is a network round-trip to a hosted model — realistically tens to hundreds of milliseconds even for a fast model, before anything else in the pipeline runs. Stripe’s public description of Radar (via a third-party engineering breakdown — a secondhand account, not Stripe’s own documentation) describes their core decision completing in under 100 milliseconds while evaluating 1,000+ signals, and that’s a tightly optimized, purpose-built scoring path, not a general-purpose LLM call. Making a language model the primary, synchronous score on every transaction inside a 200ms all-in budget isn’t a model-quality question, it’s arithmetic. The budget has no room for it as the only signal.
XGBoost as the baseline score, trained on engineered tabular and behavioral features, kept as the primary and fastest signal in the pipeline. Where the LLM actually fits is a separate decision (§4) — it doesn’t replace the tree-based baseline, it runs alongside it.
Why query-time joins were the fastest thing to eliminate: the features that actually catch fraud — "how many transactions has this card attempted in the last 10 minutes," "has this device fingerprint been seen with three different cardholders this week" — are rolling-window aggregates over recent history. Computing those as ad-hoc joins at scoring time, inside a sub-200ms budget that also has to cover XGBoost inference, the LLM call, and network round-trips, means running non-trivial aggregation queries under time pressure on every transaction. That’s not a latency optimization to solve later; it’s the wrong place in the pipeline to do the work at all.
Why nightly batch tables were the second thing to go, for a reason specific to fraud rather than generic staleness: fraud specifically targets the gap. An attacker running a card-testing script against a stolen card list generates a burst of transactions in minutes; a velocity feature refreshed once a night has already been irrelevant for the entire attack window by the time it updates. Features reflecting live behavior, not a snapshot, is close to the entire point of adding behavioral signals at all.
Why a real-time feature store, specifically: Feast’s own documentation on online-serving performance describes a Java gRPC server backed by Redis as their best-performing configuration, reporting 4–10x better throughput than alternative online stores in their own benchmarking — Feast’s self-reported number, not independently audited, but consistent with Redis’s in-memory read profile. The property that mattered more than the multiplier, though, is training-serving consistency: Feast (and comparable platforms like Tecton) define a feature transformation once and use the same definition for both the offline store (training) and the online store (live predictions). For a regulated model where "why did the model score this transaction as risky" has to be answerable, guaranteeing the training-time and serving-time feature are computed identically closes off an entire category of "the model behaves differently in production than validation" bugs that are notoriously hard to diagnose after the fact.
A Feast-style real-time feature store, with streaming writes from the Kafka event log (§3) populating a Redis-backed online store, giving XGBoost single-digit-millisecond reads on freshly computed behavioral and velocity features at scoring time.
Why this is a genuinely different calculus than a typical "do we need Kafka" decision: the standard case against reaching for Kafka is that most event-driven workloads are actually low-volume, ephemeral notifications — for those, a lightweight queue is correctly the right call, and Confluent’s own guidance on when not to use Kafka makes exactly that point. This isn’t that workload: every card-not-present transaction the client processes generates an event, continuously, at meaningful volume — not the "a few hundred background jobs a day" shape that rules Kafka out elsewhere.
The requirement that actually decided it wasn’t throughput — it was durable replay: SQS-style queues and RabbitMQ are built around delivery — a message is consumed, acknowledged, and, without extra tooling bolted on, gone. Kafka’s core design property, documented in Apache’s own architecture docs, is the opposite: an append-only log consumers read without removing, retained for a configured period, replayable by any number of independent consumer groups. For a regulated fraud pipeline, a compliance reviewer asking "reconstruct exactly what the system knew about this transaction six weeks ago" is a routine request, and a transient queue architecturally can’t answer it without a second system bolted on to fake retention.
Why the throughput headroom was worth citing even though it wasn’t the deciding factor: Confluent’s own published benchmarks report Kafka clusters sustaining roughly 2 million writes/second on a 3-broker cluster, and tens of millions of messages/second in consumer-heavy scenarios with sub-5-millisecond median end-to-end latency. At the far end of the spectrum, LinkedIn’s engineering blog describes running Kafka at roughly 7 trillion messages/day across 100+ clusters — cited explicitly as an extreme-scale reference, not a baseline expectation; a mid-market payments processor isn’t running at LinkedIn’s scale. The honest version: Kafka has throughput headroom orders of magnitude beyond what this workload needs, which is nice to have, but it’s the durable-replay property, not the throughput ceiling, that actually made the decision.
Kafka, ingesting the live transaction stream once, feeding both the feature store’s streaming aggregation (§2) and long-term retention for compliance replay and future model retraining — the same event log serving multiple independent downstream consumers rather than each one needing its own delivery mechanism.
Why sequential gating (score first, reason about borderline cases second) was the natural-seeming design that didn’t fit the budget: the intuitive design runs XGBoost first, spending an LLM call only on transactions that land in an ambiguous score band — cheap, because most transactions never hit the LLM. The problem: this makes the LLM call additive on top of tree-model latency, specifically on the transactions the client cares most about getting right, and it never touches the confidently-scored transactions at all — including the false negatives, fraud that looks statistically unremarkable in engineered features but would read as suspicious from the transaction narrative itself. A design that only applies language reasoning to cases the tree model was already unsure about can’t fix the tree model’s own blind spots.
Why running XGBoost and the LLM call concurrently, not sequentially, is what makes the latency budget work at all: if the LLM call runs in parallel with tree inference rather than after it, total added latency is close to the LLM call’s own duration, not the sum of both. Tree inference itself is fast enough not to matter here — single-digit-millisecond scoring is standard industry practice for gradient-boosted models with hundreds to low-thousands of trees on CPU. That means the entire latency budget for this decision comes down to one number: how fast is the LLM call, on its own.
Why GPT-4o-mini, not GPT-4o, once the call runs on every transaction rather than a sampled subset: OpenAI’s July 18, 2024 announcement introducing GPT-4o-mini positions it explicitly for "fast, real-time text responses" and "applications that require chaining or parallelizing multiple model calls" — precisely this use case. GPT-4o-mini is priced at $0.15/1M input tokens and $0.60/1M output tokens, against GPT-4o’s launch pricing of $5.00/$15.00 — over a 30x gap, which matters more here than usual because the call touches every transaction. OpenAI reports GPT-4o-mini scoring 82.0% on MMLU (official figure); GPT-4o’s own MMLU is commonly cited around 88.7%, marked lower-confidence since we verified it through secondary sources, not OpenAI’s page directly. For a task narrower than general reasoning, mini’s real-time positioning and per-call cost mattered more than closing a general-knowledge gap this call isn’t really being asked to close.
Why not a self-hosted open-source model: this is a regulated financial workflow, and the practical question isn’t just inference cost — it’s who owns the burden of proving the model behaves safely and consistently over time. A managed frontier model API comes with a vendor’s own documented evaluation and safety testing, version pinning, and a paper trail a compliance conversation can point to. Standing up and continuously validating a fine-tuned open model for a fraud-adjacent reasoning task, on a 10-week build window with no dedicated MLOps function on this workstream, shifts that governance burden onto the client with no latency or cost benefit large enough to justify it at this volume.
GPT-4o-mini, called concurrently with the XGBoost score on every transaction — not gated behind a borderline-score threshold — reasoning over the transaction narrative and device/behavioral context, with its output combined with the tree-based score in a lightweight ensemble step before the routing decision (§5).
Why a binary threshold was the thing we were explicitly hired to move away from: the original rules engine was effectively a binary block/allow system, and its failure mode — a legitimate high-value customer declined with no path to a fast correction — was the specific problem in scope. A more accurate model behind the same binary structure still leaves zero room for "the model is 55% confident this is fraud" to resolve as anything but a guess. A more accurate score doesn’t fix a decision architecture with only two exits.
Why fully-automated-with-post-hoc-review doesn’t work for fraud specifically, even though it’s a legitimate pattern elsewhere: post-hoc review is the right call for a lot of quality and moderation problems — let the action happen, catch mistakes afterward. It doesn’t work here because the "action" is money moving. By the time an analyst reviews a transaction that already cleared, the funds have typically settled or the goods have shipped; catching the fraud after the fact turns a prevention problem into a recovery problem — categorically worse for the client and usually the cardholder.
Why the middle tier is what actually reduces false positives, not the model alone: the combined score from §4 gives three usable zones instead of one cutoff — confidently legitimate, confidently fraudulent, genuinely ambiguous. Routing only the ambiguous middle band to a human means the automated path still handles the overwhelming majority of transactions at full speed — most are not close calls — while the transactions where a wrong automated call costs the most get a human decision instead of a coin flip dressed up as a threshold.
Why the review console needed to show both signals, not just a risk score: an analyst making a fast, defensible decision on a held transaction needs to see why the system is unsure — which features drove the XGBoost score up, and what the LLM’s narrative reasoning actually said — side by side, not one opaque number. This is a compliance requirement as much as a usability one: a documented rationale for an override is what a disputes or audit process asks for later. We built this in React for the same reason we default to React for any internal, data-dense, authenticated ops tool — no bundle-size constraint, and the deepest hiring pool for whoever eventually maintains it. The more important design decision wasn’t the framework — it was writing analyst decisions back into the same Kafka log (§3) as labeled outcomes, so every human override becomes training signal instead of a judgment call that evaporates once the ticket closes.
A three-tier routing decision — auto-allow, auto-block, and a soft-hold for the ambiguous middle band routed to a real-time React-based analyst console that surfaces both the XGBoost feature attribution and the GPT-4o-mini reasoning trace, with analyst decisions feeding back as labeled data for future model retraining.
None of the three published numbers is a coincidence of the five decisions above; each is downstream of a specific pair of them:
Vendor-published fraud case studies give a useful sanity check on plausibility, with an honest caveat that these are best-case numbers, not typical averages: Feedzai’s published case studies report false-positive reductions in the 50–73% range with detection increases up to 114% for specific clients, while Stripe’s public claims describe a more conservative outcome (roughly 15% more fraud caught, no increase in false positives, for early adopters). Our client’s −40%/+22% sits comfortably inside that spread — closer to the middle than to either vendor’s most aggressive headline number.
Worth stating plainly, because no architecture is permanent:
None of these are failures of the original decision — they’re the conditions under which the same reasoning process, run again, would produce a different answer.
We’re publishing this confidence table on purpose. A client is better served by knowing which numbers we’d stand behind without hesitation and which ones we’d re-verify from the primary source before repeating in a board deck, than by a document that reads clean because we quietly smoothed over the difference.