When to Fine-Tune vs When to Prompt Better

The decision people reach for too early

"The model isn't quite right, let's fine-tune it" is a common instinct and usually the wrong first move. Fine-tune vs prompt engineering is a real fork, but most of what teams want from fine-tuning — better accuracy, fewer weird outputs — comes from a better prompt, a few examples, or grounding in retrieval, at a fraction of the cost.

What fine-tuning actually buys

• Format and behaviour consistency. The model reliably produces the structure and tone you want, without a long prompt spelling it out every call.
• Lower per-call cost. A fine-tuned smaller model can match a prompted larger one on a narrow task, and it's cheaper per call and often faster.
• A narrow task done well at volume. Classification, extraction, a specific transformation — things you can define with a dataset.

What it does not buy

New knowledge, reliably. Fine-tuning nudges behaviour; it doesn't install facts you can trust. If the model needs to know your return policy or your current catalogue, that's retrieval's job, not fine-tuning's — and a fine-tuned model still hallucinates about anything outside its training set.

Symptoms that mean "prompt better"

• The output is sometimes right — the model can do it, it's just inconsistent. Tighten the instructions, add constraints, give it 3-5 examples in the prompt.
• It gets facts wrong — add retrieval, don't fine-tune.
• It's verbose or off-tone — that's almost always prompt wording.

Symptoms that mean "consider fine-tuning"

• Prompting plateaued: you've iterated, added examples, and the last 5-10% of accuracy won't come.
• The prompt is enormous and expensive because it's carrying dozens of examples every call.
• You need a small, cheap, fast model for a task a large model does well but too slowly or too expensively at your volume.

The order to try things

Prompt engineering, then few-shot examples in the prompt, then retrieval for anything fact-dependent, then — only if those hit a ceiling and you have a real labelled dataset and an eval harness — fine-tuning. Teams that start at step four usually spend weeks proving they should have started at step one. The projects in our portfolio mostly run off-the-shelf models with strong prompting and retrieval; fine-tuning shows up where the task is bounded and runs at high volume, like a detection model trained on real store photos.

The dataset is the real gate

Fine-tuning needs hundreds to thousands of labelled examples that represent the real task — the messy inputs, the edge cases, the distribution you actually see — not twenty you wrote to illustrate the idea. If getting that dataset means a labelling project, the labelling project is now on the critical path, and it's often longer than the fine-tuning itself. No dataset, no fine-tuning — prompt better instead.

Fine-tuning has its own costs

A real labelled dataset, an eval set to know it helped, a training pipeline, and re-training as the task drifts. It's a standing commitment, not a one-time tweak. Budget for it like any other model you own.

Two quick examples

• A support agent gets intent classification right 90% of the time, inconsistently. Don't fine-tune. Add five examples of the failing intents to the prompt, re-run the eval set, and see if the last 10% comes. It usually does.
• A classification step runs on every inbound ticket at high volume, and the frontier model is accurate but too slow and too expensive at that rate. This is a fine-tuning case: a small model, fine-tuned on your labelled tickets, matching the big one on this narrow task at a fraction of the cost per call.

Where this stops being right

• A genuinely stable, narrow, high-volume task where prompting works but costs too much — fine-tuning a small model is the right call, and the ROI is clear.
• A task you can't define with a dataset — open-ended reasoning, judgement — isn't a fine-tuning candidate regardless of how much prompting frustrates you.
• Very early exploration — don't fine-tune while the task definition is still moving.

FAQ

Will fine-tuning fix hallucinations? Not the ones about your data. Fine-tuning shapes behaviour; retrieval provides facts. A fine-tuned model still invents things outside its training set.

Prompt engineering or few-shot first? Instructions first, then add 3-5 examples in the prompt. If that plateaus and the prompt is getting expensive, fine-tuning enters the conversation.

What do we need before fine-tuning? A labelled dataset that represents the real task, and a fixed evaluation set to prove the fine-tuned model actually beats the prompted one.

ISTRALLEN exhausts prompting and retrieval before fine-tuning, and only fine-tunes bounded high-volume tasks — see what we do.