When to Use an LLM and When a Classic ML Model Still Wins

The default reach isn't always the right one

The 2025 reflex is to point an LLM at every problem. On a large class of tasks a gradient-boosted tree or a small classifier is faster, cheaper, more accurate, and easier to defend to a regulator. LLM vs traditional machine learning is a question of fit, not of how modern the tool is.

Where a classic ML model wins

• Structured, tabular data with engineered features. Fraud scoring, churn, pricing, credit risk. Controlled benchmarks continue to find tree-based models state-of-the-art on medium-sized tabular data — the regime most business datasets sit in.
• Tight latency and cost budgets. A gradient-boosted model scores in single-digit milliseconds on CPU. An LLM call is a network round-trip to a hosted model — tens to hundreds of milliseconds before anything else runs.
• High-volume, well-defined classification or detection. The shelf-monitoring model in our portfolio is a trained vision detector, not an LLM, because the task is bounded and runs continuously at the edge.
• When you need a calibrated probability and a tunable threshold, not a verdict you can't move.

Where an LLM wins

• Unstructured input — free text, call transcripts, messy narratives, turning an image into a description.
• Tasks defined by instructions, not a labelled dataset. When you can't realistically get 50,000 labels, an LLM that follows a spec is the practical option.
• Reasoning over context that's different every time, where no fixed feature set captures it.
• When a human-readable rationale is part of the deliverable.

The hybrid that beats either alone

The fraud-scoring project runs both: a gradient-boosted baseline as the fast, calibrated primary signal, and an LLM reasoning layer over the transaction narrative and device signals, running concurrently and combined in a lightweight ensemble step. The tree catches what's statistically odd in the features; the LLM catches what's odd in the story. Two signal types covering each other's blind spots — a common shape once you stop treating it as an either/or.

The detail that makes the hybrid affordable is running the two calls concurrently, not in sequence. If the LLM call fires in parallel with tree inference rather than after it, the added latency is roughly the LLM call's own duration, not the sum of both — and tree inference is fast enough that it barely registers. Chain them and you've doubled your budget; parallelise them and you've spent one call's worth of time to get a second, different kind of signal.

The cost comparison isn't always what you expect

"Classic ML is cheaper" holds at volume, where a per-call LLM charge on millions of transactions adds up. It flips for a low-volume task where the real cost is the labelled dataset and the feature pipeline you'd have to build to train a classic model at all. If you need a few hundred decisions a month on messy input, a prompt against a hosted model can be genuinely cheaper than standing up a training pipeline — and you can always graduate to a trained model once the volume justifies it.

A decision guide

• Tabular, latency-critical, and you have labels → classic ML.
• Unstructured, instruction-defined, rationale needed → LLM.
• Both kinds of signal present → both, combined.

Where this stops being right

• "Classic ML is always cheaper" isn't true for a low-volume task where building the labelled dataset and feature pipeline costs more than writing a prompt.
• "LLMs can't do tabular" isn't true either — they can, just slower and less accurately than trees in the regime trees own.
• The latency argument changes if the LLM runs asynchronously, off the request's hot path. Then its round-trip cost stops being a blocker.

FAQ

Can an LLM replace our fraud or risk model? As the only synchronous score on every transaction, usually not — latency and calibration are the problems. As a reasoning layer alongside a tabular model, often yes.

Do we still need data scientists if we use LLMs? Yes — for the tabular models, the evaluation harnesses, and the judgment about which tool fits which problem.

Is fine-tuning an LLM the same as training a classic model? No. Different data requirements, different failure modes, and fine-tuning still won't give you a calibrated probability the way a trained classifier does.

ISTRALLEN picks the model class to fit the problem, not the trend — see what we do.