Fintech

What Counts as Explainable AI in Fraud Detection?

August 2026 · ISTRALLEN Team

When "the model said so" isn't an answer

A chargeback dispute lands six weeks after the transaction. The examiner's question is simple: why was this scored 0.87 and declined? "The ensemble weighted about two hundred features" is technically true and completely useless. In a regulated environment, explainable AI in fraud detection is not a research nicety — it's the line between a model you can put in front of real money and one you can't.

Explainability gets treated as a single checkbox. It isn't. A fraud decision has to be explained to at least three different audiences, each of which needs something different, and the techniques that satisfy one often don't touch the others.

A declined transaction produces one thing — a score and its per-feature attribution — but that single record has to become three different explanations, not one generic reason repeated three times. For the customer, adverse-action rules require a short, true, non-technical reason: a new device, a shipping address not linked to the account, unusual purchase velocity. For the fraud analyst working a review queue in seconds per case, the same record becomes ranked contributing signals with their actual values attached, because attribution without values just relocates the guesswork. For the compliance reviewer who shows up months later, the requirement is different again: the decision has to be reconstructable — the same transaction, the same model version, the same feature values as of that moment, the same score. The dashed line is the LLM-layer wrinkle: a narrative rationale is tempting to hand an analyst, but it only earns its place if it stays grounded in the same feature contributions the score came from and is logged with the model version — a fluent explanation that isn't faithful to the math is a liability on a money decision, not a shortcut.

Interpretable vs explainable

Worth separating two things people use interchangeably:

Interpretable model — you can read the logic directly. A logistic regression with ten features, a short decision list, a scorecard. The model is the explanation.
Explainable model — a black box (gradient-boosted trees, a neural net, an ensemble) plus a post-hoc method that attributes the output to inputs: SHAP, counterfactuals, surrogate models.

Fraud teams mostly live in the second world, because the accuracy gap between a hand-readable model and a boosted ensemble is measured in real losses and real false positives on good customers. So the practical question isn't "interpretable or not" — it's "what post-hoc explanation is faithful enough to defend."

What a fraud decision actually has to explain

To the customer (adverse action). If the decision affects access to credit, US regulation — ECOA's Regulation B, and the FCRA where a consumer report is involved — requires specific, accurate principal reasons, not "you failed our risk check." A plain card decline carries lighter requirements, but a dispute still needs a coherent rationale. The output here is a short, true, non-technical reason: new device, shipping address not linked to this account, unusual purchase velocity.

To the fraud analyst. Someone is working a review queue with seconds per case. They need the top contributing signals, ranked, with values attached: device age 2 hours; amount 4.1x this account's 90-day median; billing country not equal to IP country. Attribution without the underlying values just relocates the guesswork.

To the compliance reviewer, months later. This one is a data problem disguised as an ML problem. "Reconstructable" means: same transaction, same model version, same feature values as of that moment, same score. If your feature store has since updated, or you can't pin which model version scored it, no explanation method saves you. On our fintech fraud-scoring project the hard requirement wasn't accuracy — it was that every decision stay reconstructable after the fact for a compliance reviewer, and that drove as much of the architecture as the model did.

Techniques that actually get used

Reason codes. A fixed vocabulary of decline reasons, mapped from feature contributions to a human category. Unglamorous, decades old, and still the backbone of adverse-action notices because they're stable and auditable.

SHAP values. Per-transaction, additive feature attribution; fast for tree models via TreeSHAP. Good for analyst tooling and for deriving reason codes. Caveat: when features are correlated — and fraud features always are — SHAP splits credit between them in ways that can look arbitrary. It explains the model's behavior, not the underlying reality.

Counterfactuals. "This would have passed if the amount were under $120 and the device were older than 30 days." Intuitive for humans and useful for analysts. Expensive to compute well, and risky to expose externally — it hands an attacker the boundary.

Monotonic constraints. Underrated. Force the model so that, for example, higher velocity never decreases risk. You give up a little fit and gain a model whose behaviour is defensible by construction — you can promise a direction of effect without running an explainer at all.

The LLM-layer wrinkle

A pattern some teams use — and one we shipped on that fraud engagement — is a gradient-boosted baseline plus an LLM reasoning layer over transaction narratives and device signals. The LLM can produce a fluent rationale, which is tempting to hand straight to an analyst or drop in a case file.

The trap: a fluent explanation is not automatically a faithful one. An LLM will happily narrate a plausible story that has nothing to do with what moved the score. If you use a reasoning layer, its narrative has to be anchored to the same structured feature contributions the score came from, and logged with the model version, so a reviewer can check the story against the math. An ungrounded LLM rationale on a money decision is a liability, not a feature.

Where this stops being right

Explainability has a running cost. Computing and storing attribution for every transaction, plus full audit retention, inside a sub-200ms budget is real infrastructure. If you're not regulated and disputes are rare, structured logging of inputs and score may be enough — don't build the audit cathedral you don't need.
Explanations help attackers. Every reason you expose externally is a probe into the boundary. Customer-facing reasons should be true but coarse; detailed attribution stays internal.
"Explainable" is a constraint, not a free win. A fully interpretable model that catches materially less fraud costs you in losses and in false positives on legitimate customers. You optimise under the explainability constraint; you don't get it for nothing.
Explanations drift with the model. A post-hoc explainer on a model whose data has shifted produces confident, stale reasons. Explanation quality needs monitoring the same way score calibration does.

FAQ

Is a logistic regression "explainable" enough by itself? Its coefficients are readable, but on a few hundred engineered features "readable" and "understood" aren't the same thing. You still need a reason-code mapping for anyone downstream.

Does using SHAP satisfy regulators? No framework names SHAP. Regulators want specific, accurate principal reasons and a decision you can reconstruct. SHAP is one defensible way to derive those reasons — not a compliance checkbox on its own.

Can an LLM just write the explanation for us? Only if it's grounded in the actual feature contributions and logged with the model version. A free-text rationale that isn't tied back to what drove the score is a risk, not a shortcut.

ISTRALLEN builds fraud models for regulated fintech teams where every score has to survive an audit — that approach is laid out under AI for Fintech.

See it in production

AI for Fintech → Fraud-scoring case study →

← All articles