How to Keep an LLM Feature's Cost Predictable as You Scale

The bill scales with success

An LLM feature that costs little in a pilot costs a lot at production volume, and the growth isn't linear with users — multi-step agents make one task several model calls, and traffic spikes multiply the bill exactly when you're leaning on the feature. LLM cost control is the set of levers that keep that from becoming a surprise.

Route by difficulty

The biggest lever. Not every call needs a frontier model. A small, fast model can be an order of magnitude cheaper per call, and on many tasks the quality difference is negligible. On our fraud-scoring engagement the reasoning layer uses a lightweight model precisely because the call runs on every transaction — the cost of the frontier model there was arithmetic, not a quality question. Classify the incoming task and send the easy ones to the cheap model.

Cache the head of the distribution

A large share of real traffic is the same few hundred queries. Cache the model's output (or the retrieval plus the answer) for those, keyed on a normalised query, and serve them without a model call. It cuts the bill and the latency together.

Cap calls per task

A runaway agent can loop, retry, and chain calls indefinitely. Put a hard ceiling on model calls per task, and a timeout on the whole interaction. When it hits the ceiling, escalate — a task that needed twelve model calls was probably one a human should have seen.

Budget for the peak multiplier

Size the monthly cost against peak-month volume, not the average. A support agent at Black Friday traffic, a fraud pipeline during an attack wave — these are forecastable, and the per-call cost times the peak volume is the number to plan against.

Set a budget per task and enforce it

Give each task a soft cost budget. When a task exceeds it — an unusually long conversation, a retry storm — that's a signal, and often an escalation: a case that cost 4x the norm to process is a case a human should probably have seen. Enforcing the budget also caps your worst-case exposure to a prompt-injection or a loop that would otherwise run unbounded.

Monitor it like a metric

Cost per task, calls per task, and the small-model routing rate belong on a dashboard with alert thresholds. A quiet drift — the router sending more traffic to the frontier model, the average calls-per-task creeping up — should page someone before the invoice does. Each project in our portfolio carries a per-unit run cost — per transaction, per conversation, per search — sized against real volume before the build, so the monthly number is a forecast, not a surprise.

Where the cost hides

The line teams underestimate is the multi-call reality of agents and tool use. "One support conversation" is often four to eight model calls once you count the tool round-trips; "one fraud decision" might be one call but on every transaction. Cost the task, not the call, and know your calls-per-task before you sign off a budget.

Where this stops being right

• Low volume — the fixed infrastructure dominates and the per-call optimisation isn't worth the complexity.
• A task where quality genuinely can't tolerate the small model — measure it against your eval set before assuming the cheap model is fine.
• Estimates age — re-quote per-token pricing before committing; it moves, usually down, and a stale quote is a different number.

FAQ

What's the single biggest cost lever? Routing the calls that don't need a frontier model to a small one. On a per-call feature it can cut the model bill by an order of magnitude.

Does caching actually help for search or support? Yes — the head of real query distributions is surprisingly concentrated. Caching the common queries removes a large fraction of model calls. Normalise the key (lowercase, trim, collapse whitespace) so near-identical queries hit the same cache entry, and set a TTL short enough that a stale answer can't outlive a policy change.

How do we stop an agent from running up the bill? A hard cap on model calls per task and a timeout on the interaction, with escalation to a human when either is hit.

ISTRALLEN builds LLM features with routing, caching, and per-task caps so the cost stays predictable at scale — see what we do.