When to Fine-Tune vs When to Prompt Better

The decision people reach for too early

"The model isn't quite right, let's fine-tune it" is a common instinct and usually the wrong first move. Fine-tune vs prompt engineering is a real fork, but most of what teams want from fine-tuning (better accuracy, fewer weird outputs) comes from a better prompt, a few examples, or grounding in retrieval, at a fraction of the cost.

What fine-tuning actually buys

• Format and behaviour consistency. The model reliably produces the structure and tone you want, without a long prompt spelling it out every call.
• Lower per-call cost. A fine-tuned smaller model can match a prompted larger one on a narrow task, and it's cheaper per call and often faster.
• A narrow task done well at volume. Classification, extraction, a specific transformation: things you can define with a dataset.

What it does not buy

New knowledge, reliably. Fine-tuning nudges behaviour; it doesn't install facts you can trust. If the model needs to know your return policy or your current catalogue, that's retrieval's job, not fine-tuning's, and a fine-tuned model still hallucinates about anything outside its training set.

Symptoms that mean "prompt better"

• The output is sometimes right: the model can do it, it's just inconsistent. Tighten the instructions, add constraints, give it 3-5 examples in the prompt.
• It gets facts wrong: add retrieval, don't fine-tune.
• It's verbose or off-tone: that's almost always prompt wording.

Symptoms that mean "consider fine-tuning"

• Prompting plateaued: you've iterated, added examples, and the last 5-10% of accuracy won't come.
• The prompt is enormous and expensive because it's carrying dozens of examples every call.
• You need a small, cheap, fast model for a task a large model does well but too slowly or too expensively at your volume.

The order to try things

Prompt engineering, then few-shot examples in the prompt, then retrieval for anything fact-dependent, then, only if those hit a ceiling and you have a real labelled dataset and an eval harness, fine-tuning. Teams that start at step four usually spend weeks proving they should have started at step one. The projects in our portfolio mostly run off-the-shelf models with strong prompting and retrieval; fine-tuning shows up where the task is bounded and runs at high volume, like a detection model trained on real store photos.

The dataset is the real gate

Fine-tuning needs hundreds to thousands of labelled examples that represent the real task (the messy inputs, the edge cases, the distribution you actually see) not twenty you wrote to illustrate the idea. If getting that dataset means a labelling project, the labelling project is now on the critical path, and it's often longer than the fine-tuning itself. No dataset, no fine-tuning: prompt better instead.

Fine-tuning has its own costs

A real labelled dataset, an eval set to know it helped, a training pipeline, and re-training as the task drifts. It's a standing commitment, not a one-time tweak. Budget for it like any other model you own.

Two quick examples

• A support agent gets intent classification right 90% of the time, inconsistently. Don't fine-tune. Add five examples of the failing intents to the prompt, re-run the eval set, and see if the last 10% comes. It usually does.
• A classification step runs on every inbound ticket at high volume, and the frontier model is accurate but too slow and too expensive at that rate. This is a fine-tuning case: a small model, fine-tuned on your labelled tickets, matching the big one on this narrow task at a fraction of the cost per call.

Where this stops being right

• A genuinely stable, narrow, high-volume task where prompting works but costs too much: fine-tuning a small model is the right call, and the ROI is clear.
• A task you can't define with a dataset: open-ended reasoning, judgement, isn't a fine-tuning candidate regardless of how much prompting frustrates you.
• Very early exploration: don't fine-tune while the task definition is still moving.

FAQ

Will fine-tuning fix hallucinations? Not the ones about your data. Fine-tuning shapes behaviour; retrieval provides facts. A fine-tuned model still invents things outside its training set.

Prompt engineering or few-shot first? Instructions first, then add 3-5 examples in the prompt. If that plateaus and the prompt is getting expensive, fine-tuning enters the conversation.

What do we need before fine-tuning? A labelled dataset that represents the real task, and a fixed evaluation set to prove the fine-tuned model actually beats the prompted one.

ISTRALLEN exhausts prompting and retrieval before fine-tuning, and only fine-tunes bounded high-volume tasks. See what we do.