Point-in-Time Correctness: Building a Fraud Training Set That Matches Production

The bug that makes a model look better than it is

A fraud model validates beautifully offline and then underperforms in production. The usual cause isn't the model — it's the training set. Point-in-time correctness in a feature store is the property that stops it: every training row uses feature values as they were at the moment of that decision, not as they are now.

What leakage looks like

Take a feature like "number of transactions on this card in the last 24 hours." If you compute it by querying a table that has been updated since, the count for a transaction from last month now includes transactions that happened after it. The model learns from information that didn't exist when the real decision had to be made. It's learning from the future.

At scoring time in production, that future doesn't exist. So the model relies on a signal it can't actually get, and its live performance drops to match reality.

How you build a point-in-time-correct set

• Time-versioned features. Either a feature store that supports time-travel, or an append-only event log you replay to reconstruct state at any past moment.
• An as-of join. For each labelled transaction, join feature values with a cutoff at that transaction's timestamp — nothing after it counts.
• A cutoff per row, not a global one. Each training example has its own "as of" moment.

On our fraud-scoring project, every decision has to be reconstructable months later for a compliance reviewer, and the same event log that makes that possible is what lets training sets be assembled with feature values frozen at decision time.

The train/serve skew connection

Point-in-time correctness is one half of matching training to production. The other half is identical feature definitions — the code that computes "24-hour velocity" in training has to be the same code that computes it in serving. A feature store gives you both: time-travel for the first, a single definition for the second.

The as-of join, concretely

Take a training row for transaction T, which was scored at time t0. Its features are computed as: transaction count for that card where the transaction timestamp is strictly before t0; "device seen before" as a boolean evaluated against events prior to t0; amount-versus-90-day-median using only history up to t0. The label is the chargeback that was filed at t0 + 45 days. Every feature has the same cutoff; the label comes from the future because by training time the future has arrived. Nothing computed after t0 is allowed into the feature columns.

Split by time, not at random

A random train/test split shuffles rows from every period together, so the test set contains transactions from before some training transactions — the model is graded partly on a past it was allowed to learn from. Split by time instead: train on an earlier window, test on a later one. It's a harder test, and it's the one that matches how the model will actually be used.

The label-timing wrinkle

Fraud labels arrive weeks later, through chargebacks and confirmed-fraud reports. That's fine — the label for a training row is "known now." It's the features that must be "as of then." Don't confuse the two timelines.

Where this stops being right

• No event log or time-versioned features — you can't do true point-in-time joins; a snapshot-based approach with a documented approximation is the fallback, and you note the risk.
• A non-adversarial, low-stakes problem where a small leak doesn't materially move the model.
• Very frequent retraining on recent data shrinks the leak window — it doesn't close it, but it reduces the damage.

FAQ

What's the symptom of a point-in-time bug? Offline metrics markedly better than online performance, with no other explanation. That gap is almost always leakage.

Do we need a feature store to do this? You need time-versioned features. A feature store is the common way; an append-only event log you replay is another.

Does this matter outside fraud? Any model with time-based features and delayed labels — churn, credit risk, demand forecasting — has the same trap. Fraud just makes it expensive fastest, because an adversary is actively probing the gap between your model and reality.

ISTRALLEN builds fraud training pipelines with point-in-time-correct feature joins so offline validation predicts production; see AI for Fintech.