Measuring Whether an AI Feature Actually Moved the Business Metric

The number nobody agreed on

An AI feature gets approved on "it'll help" and reviewed a year later against a metric nobody defined. AI feature ROI measurement is genuinely doable — but only if you baseline the right thing before you ship, because you can't reconstruct the baseline afterward.

Use a holdout if you possibly can

A slice of traffic keeps the old process over the same period. A slice of stores stays on manual audits. A share of transactions scores on the old model. This controls for seasonality, promotions, and demand spikes automatically — the holdout and the treatment group experience the same conditions. It's worth real effort to arrange.

If you can't split traffic

Pre/post works, but annotate it: mark known seasonal effects, promo calendars, traffic-mix changes, and anything else that moved. And don't read the result until a full outcome cycle has closed — chargebacks lag 30 to 90 days, so a week-two fraud readout is noise.

Define the metric and its owner before launch

Write down the business metric — search-session conversion, first-response time, on-shelf availability, call-centre load — the measurement window, and the source of each figure. Assign an owner per input. On a cross-team number, if risk, growth, and the platform team don't sign off on the method before launch, the readout becomes an argument about methodology instead of a decision.

The traps that inflate the number

• Deflection counted as resolution. A conversation the agent "closed" that the customer re-opened tomorrow isn't a win. Net out re-contacts.
• Attribution to the wrong scope. Crediting sitewide conversion to a search change — measure the search segment.
• Ignoring the run cost. Model inference, infra, and the escalation staffing that never goes to zero are the denominator.
• Taking a vendor's headline as your forecast. Use published outcomes as a sanity range, not a target.

The structure

net = (business value from the moved metric) − (build cost, amortised) − (monthly run cost)

The projects in our portfolio each report a specific moved metric — search conversion, response time, product availability, call-centre load — measured against a baseline, not asserted. Payback is then build cost divided by monthly net, measured across whatever cycle the value accrues in — peaks, quarters, chargeback windows — not weeks.

Baseline before you ship — you can't do it after

The things you need from before the feature existed can't be reconstructed later: the old conversion rate, the old first-response time, the old audit hours per store, the old decline-and-false-positive rate. For a few weeks before launch, log them deliberately. A feature that shipped without a baseline can only ever be evaluated on anecdote.

The soft buckets still count

Some value isn't in the headline metric. A support agent that answers instantly instead of after 20 minutes retains some carts that would have been abandoned — that's revenue, and it's often the biggest bucket in peak season, even though it's harder to attribute. Bracket it with a range and state the assumption out loud; a conservative estimate that survives the finance review beats an aggressive one that gets picked apart.

Where this stops being right

• A holdout genuinely isn't possible (a single-instance system, a small store count) — carry the result as a range and state the assumptions.
• The value is mostly qualitative (a better experience with no clean metric) — say so rather than inventing a number.
• The feature is cheap and obviously positive — don't spend more measuring it than it costs to run.

FAQ

Holdout or pre/post? Holdout whenever you can split traffic — it controls for seasonality for free. Pre/post works but needs explicit adjustment for anything that moved during the window, and it's weaker because you're comparing two different time periods, not two groups in the same one.

How long before we can read the ROI? Long enough for a full outcome cycle — a chargeback window, a restock cycle, a promo season — plus the follow-up window so re-contacts are counted.

What's the most common inflation? Counting deflection as resolution, and ignoring the run cost. Net out the re-contacts and subtract what it costs to operate — model inference, infra, and the escalation staffing that never reaches zero.

ISTRALLEN builds AI features instrumented against a holdout so the business effect is measured, not assumed — see what we do.