Prompt20
NewsDataBlog
All posts
fine-tuningloraqloratraining-datamodel-customizationhow-toevergreen

How to Fine-Tune an LLM (and When You Shouldn't)

The decision tree first — prompt vs RAG vs fine-tune — then the practitioner mechanics: LoRA and QLoRA, building a dataset, evaluating results, and the failure modes that quietly waste your GPU budget.

By Prompt20 Editorial · 24 min read

Here is the uncomfortable truth most fine-tuning tutorials skip: the majority of teams that set out to fine-tune an LLM should not. Not because it's hard — the tooling is genuinely good now — but because they're reaching for a training run to solve a problem that a better prompt, a few examples, or a retrieval system would solve faster, cheaper, and with less to maintain. Fine-tuning is a real tool with a narrow, valuable job. The skill isn't running the training loop. The skill is knowing when the training loop is the right answer.

So this guide leads with the decision, not the code. First we'll figure out whether you should fine-tune at all. Then, assuming you should, we'll walk the actual mechanics — LoRA and QLoRA, dataset construction, evaluation, and the quiet failure modes that turn a weekend project into a month of confused GPU spend. The goal is that you finish able to make the call confidently in either direction, because "we decided not to fine-tune, and here's why" is often the most senior thing an engineer says all quarter.

Table of contents

Key takeaways

  • Fine-tuning changes behavior, not knowledge. It's the wrong tool for "teach the model new facts" and the right tool for "make the model reliably respond in this format / style / structure." Use retrieval for facts.
  • Try the cheap options first, in order: prompt engineering → few-shot examples → retrieval (RAG) → fine-tuning. Each step up costs more to build and maintain. Most problems stop before the last step.
  • You almost never do full fine-tuning. Parameter-efficient methods — LoRA and its memory-frugal cousin QLoRA — train a tiny fraction of the weights, fit on modest hardware, and are far easier to version and roll back.
  • The dataset is the whole game. A few hundred to a few thousand high-quality, consistent examples beat tens of thousands of noisy ones. Garbage-in shows up as confident, fluent wrongness.
  • If you can't measure it, don't train it. Build the evaluation set before you fine-tune, so you can prove the tuned model is actually better and not just different.
  • The classic failure modes are boring: inconsistent labels, train/serve prompt mismatch, catastrophic forgetting, and overfitting to a narrow style. None are exotic; all are avoidable.

What fine-tuning actually does

A base or instruction-tuned model is a fixed set of weights that encodes a general policy: given this input, produce that kind of output. Fine-tuning continues training on your examples, nudging those weights so the model's default behavior shifts toward the patterns in your data. That's it. You are not "adding a database." You are adjusting a reflex.

This distinction is the single most important idea in the whole topic, so hold onto it: fine-tuning teaches form far more reliably than it teaches fact. If you show a model 800 examples of your support tickets being classified into your taxonomy, it gets genuinely good at emitting your taxonomy in your format. If you show it 800 documents hoping it will "learn your product," you'll get a model that has vaguely absorbed some vocabulary and will still hallucinate specifics — because facts you show a handful of times don't reliably stick, and the ones that do can't be updated without retraining. The moment a fact changes, a fine-tuned fact is stale. A retrieved fact is current.

So the honest one-line spec for fine-tuning is: use it to change how the model behaves, when the behavior you want is hard to specify in a prompt but easy to demonstrate with examples.

To make that concrete, look at the mechanism. A language model is trained on one objective: predict the next token given the tokens so far. During fine-tuning you feed it your example — say, a support ticket followed by the correct label — and at each position you compare the token the model would have predicted against the token that actually appears in your target. The gap between those two is the loss. Gradient descent then nudges the weights a small step in the direction that would have made your target token slightly more probable next time. Repeat that across a few hundred examples for a few passes, and the model's default probabilities shift toward your patterns. Nothing about this process "stores a fact" in a retrievable slot. It reshapes a probability distribution over tokens. That is precisely why form transfers well (form is a stable, repeated statistical pattern across your examples) and fact transfers badly (a specific fact appears in a handful of examples, contributes a tiny gradient signal, and gets averaged against everything else the weights already encode).

A useful mental image: the pretrained model is a landscape of behaviors carved by trillions of tokens. Fine-tuning does not rebuild the landscape; it applies gentle, localized pressure to tilt the paths water tends to flow down. Push too hard and you don't just deepen your desired channel — you erode the surrounding terrain, which is the mechanism behind catastrophic forgetting that we'll return to later. The gentleness is not timidity; it is the correct dose. Most fine-tuning failures are an overdose: too many epochs, too high a learning rate, too little data diversity, all of which press so hard that the model forgets how to be a general model in exchange for memorizing your training set verbatim.

The decision tree: should you even fine-tune?

Walk these in order. Stop at the first one that solves your problem.

1. Prompting. Can you get the behavior by writing better instructions? A surprising amount of "we need a custom model" is really "we wrote a vague prompt." Be specific, give the model a role, show the output schema, state the constraints. If you haven't seriously invested here, you're not ready to fine-tune. (See how to write better prompts.)

2. Few-shot examples. Can you get it by putting 3–10 examples in the prompt? In-context examples are the fastest way to teach format and tone. They cost tokens per request, but they cost you zero training infrastructure and change instantly when you edit them. If five examples in the context window fix it, you're done.

3. Retrieval (RAG). Is the real problem "the model doesn't know my data"? Then you want retrieval, not fine-tuning. Put the facts in a store, fetch the relevant ones at query time, and hand them to the model. This keeps knowledge current and auditable. See RAG in production and the embeddings and vector search guide for how that's built.

4. Fine-tuning. You reach here when: the behavior is consistent and demonstrable, prompting plus few-shot can't hit your reliability bar, the task is high-volume enough that shorter prompts save real money, or you need the model to internalize a style/format so thoroughly that examples in every request is wasteful. Now it's the right tool.

A quick way to feel the boundary:

Symptom Reach for
"It doesn't follow our output format reliably" Fine-tune (or stronger few-shot)
"It doesn't know our internal docs / latest data" Retrieval, not fine-tuning
"It's too chatty / wrong tone / wrong persona" Prompt first, fine-tune if persistent
"We need a smaller/cheaper model to match a big one on ONE task" Fine-tune (distillation-style)
"It hallucinates specific facts" Retrieval + guardrails, not fine-tuning
"We need it to reason through novel problems better" Usually a better base model, not fine-tuning

Notice the pattern: fine-tuning wins on narrow, repeatable behavior. It loses on knowledge and on general capability. If your instinct is "we'll fine-tune it to be smarter," stop — you want a better base model, and possibly the open-weights guide to pick one. This ladder is a specialization of a more general build-vs-buy question; the how to choose an LLM for your app guide walks the same tradeoffs from the model-selection side.

Fine-tune vs RAG vs prompting, quantified

The decision tree above sorts by problem type. It's worth also sorting by cost, latency, and maintenance, because those are what actually get you overruled in a design review. Here is the same three options compared on the axes that matter operationally.

Axis Prompting / few-shot Retrieval (RAG) Fine-tuning
What it changes Nothing in the model; instructions per request What the model sees at inference The model's default behavior
Good for Format, tone, simple tasks Facts, fresh data, citations Deeply ingrained style/format, token savings
Time to first result Minutes Days Days to weeks
Ongoing maintenance Edit a string Keep the index fresh Retrain to change behavior
Cost profile Higher per-request tokens Retrieval infra + tokens Training cost once, cheaper per request
How you update it Change the prompt Update documents Rebuild the dataset and retrain
Auditability High (prompt is visible) High (sources are cited) Low (behavior is baked in)
Fails by Vague instructions Bad retrieval / stale index Wrong data / forgetting

The important insight buried in that table: these are not mutually exclusive, and the strong systems use all three. A production assistant might use a fine-tuned model (so every response follows house style and structure without a giant system prompt), fed by retrieval (so it answers from current documents it can cite), steered by a compact prompt (so per-request behavior is still tunable without retraining). Fine-tuning and RAG are complements, not rivals — one shapes behavior, the other supplies knowledge. The teams that frame it as "fine-tune or RAG" have usually mistaken a knowledge problem for a behavior problem or vice versa.

A blunt heuristic for the common case: if the answer changes when your documents change, you need retrieval. If the answer changes when your format or policy changes, you need a prompt or a fine-tune. If you're editing the same 40-line system prompt into every request and paying for those tokens millions of times, that's the signal that a fine-tune could pay for itself — you're renting per request what you could own once.

The tuning taxonomy: SFT, preference tuning, continued pretraining

"Fine-tuning" is a suitcase word. It covers at least three distinct procedures that change different things about the model, cost different amounts, and answer different questions. Confusing them is a common way to reach for the wrong one. Here is the map.

Supervised fine-tuning (SFT) is what almost everyone means when they say "fine-tune," and what the rest of this guide focuses on. You show the model input/output pairs — a prompt and the ideal completion — and train it to reproduce the completion. It teaches the model what a good answer looks like by imitation. SFT is the workhorse: format enforcement, style, task specialization, distilling a big model's behavior into a small one. It is the cheapest and most predictable of the three, and it is almost always where you should start. LoRA and QLoRA, discussed next, are how you run SFT efficiently; they are methods, not a separate objective.

Preference tuning (DPO, RLHF, and relatives) answers a different question: not "what is the ideal answer?" but "given two answers, which is better?" Instead of a single target, you provide pairs — a preferred response and a rejected one — and the model learns to shift probability toward the kind of output humans prefer. Reinforcement Learning from Human Feedback (RLHF) does this with a separate reward model and a reinforcement-learning loop; it is powerful and how the major chat models were aligned, but it is heavy machinery — a reward model to train, a delicate RL loop to stabilize, and real expertise to run. Direct Preference Optimization (DPO) is the pragmatic modern default: it achieves a similar effect by optimizing directly on preference pairs with a simpler, supervised-style loss and no separate reward model. Reach for preference tuning when the problem is subtle quality — helpfulness, safety, "sound more like our senior agent and less like a robot" — where you can more easily say which of two outputs is better than write the single perfect one from scratch. The practical sequence is SFT first to establish competence and format, then DPO on top to polish quality. Preference tuning is a refinement layer, not a starting point.

Continued pretraining (a.k.a. domain-adaptive pretraining) is the odd one out. Here you keep training the base model on the same next-token objective it was pretrained with, but on a large corpus of raw domain text — millions of tokens of legal filings, medical notes, or code in a niche language. You are not teaching a task; you are marinating the model in a domain's vocabulary and patterns so its base competence in that domain rises. This is the closest any tuning method comes to "teaching knowledge," and even here the effect is diffuse fluency, not retrievable facts. It is expensive, data-hungry (think gigabytes of text, not hundreds of examples), and rarely what an application team needs. It matters when your domain's language is genuinely far from the model's training distribution and no amount of retrieval fixes the base model's unfamiliarity with the style of the material. For most teams, this is a lab-scale activity you consume via a domain-adapted base model rather than run yourself.

Method Data shape What it changes When to use it
SFT input → ideal output pairs Imitates good answers Default; format, style, task specialization
Preference (DPO/RLHF) preferred vs rejected pairs Shifts toward preferred quality Polish helpfulness/safety after SFT
Continued pretraining raw domain text Diffuse domain fluency Domain far from base distribution; lab-scale

The mental model: continued pretraining broadens the base, SFT specializes it to a task, preference tuning refines the task's quality. They stack in that order, and most application teams only ever need the middle one. If you're unsure which you need, you almost certainly need SFT — the other two are refinements you add once SFT has proven the concept and hit a specific ceiling that SFT alone can't clear.

LoRA and QLoRA: the methods you'll actually use

Full fine-tuning updates every weight in the model. For anything past a small model, that means a lot of GPU memory (you hold the weights, their gradients, and optimizer state), a full-size checkpoint per version, and a real risk of degrading the model's general abilities. Almost nobody outside well-funded labs does this, and you probably shouldn't either.

LoRA (Low-Rank Adaptation) is the standard alternative. Instead of editing the original weights, you freeze them and train small "adapter" matrices that get added alongside. The insight is that the change you need is low-rank — it can be represented by a couple of skinny matrices with far fewer parameters than the full layer. You end up training well under 1% of the parameters. The base model is untouched; your fine-tune is a small adapter file measured in megabytes, not gigabytes.

That small-adapter property pays off everywhere downstream:

  • Cheap to train. Fits on modest GPUs because you only store optimizer state for the adapter.
  • Cheap to store and version. Each experiment is a small file. Keeping ten variants is trivial.
  • Easy to roll back and swap. The base is constant; you can hot-swap adapters, even serve many adapters over one shared base model (see multi-tenant LoRA serving).
  • Safer. Because the base weights are frozen, you're less likely to nuke the model's general competence.

QLoRA is LoRA plus quantization: you load the frozen base model in a compressed 4-bit form to slash memory, then train the LoRA adapters on top in higher precision. The result is that you can fine-tune surprisingly large models on a single consumer-ish GPU. The trade-off is a small quality cost from quantization and somewhat slower steps, but for most practical tasks it's an excellent deal — QLoRA is what makes "fine-tune a big model on one GPU" possible. If quantization is new to you, the quantization trade-offs post explains what precision you're giving up and why it usually doesn't matter for inference-shaped work.

The low-rank math: why a tiny adapter is enough

LoRA sounds almost too good — train under 1% of the parameters and get most of the benefit of full fine-tuning? The reason it works is a genuine property of these models, worth understanding because it explains both why LoRA succeeds and when it doesn't.

A weight matrix in a transformer layer might be, say, 4096 by 4096 — roughly 16.8 million numbers. Full fine-tuning would update all of them. LoRA's bet is that the change you need to make to that matrix during fine-tuning has low "intrinsic rank" — meaning the update, however large the matrix, can be well approximated by the product of two much skinnier matrices. Instead of learning a full 4096×4096 update, you learn a 4096×r matrix (call it A) and an r×4096 matrix (call it B), where r, the rank, is small — commonly 8, 16, or 32. Their product BA has the same 4096×4096 shape as the layer, so it can be added straight onto the frozen weights, but it's constructed from only 2 × 4096 × r parameters. At r = 8, that's about 65,000 numbers instead of 16.8 million — a ~250× reduction for that layer, and you only apply it to selected layers (typically the attention projections).

Why is the update low-rank in the first place? Empirically, adapting a huge pretrained model to a specific downstream task turns out to require far less "new information" than the model already contains. The model has already learned the hard, general structure of language; your task is a comparatively small rotation of that existing capability. Research on the "intrinsic dimension" of fine-tuning found that models can be adapted to tasks by optimizing in a surprisingly tiny subspace — LoRA is the practical exploitation of that observation. You are not teaching the model to think; you are supplying a compact steering signal to competence it already has.

This also tells you LoRA's limits. If a task genuinely requires the model to learn something structurally new and broad — the continued-pretraining case above, adapting to a wildly out-of-distribution domain — a low-rank update may be too small a lever, and you'll see it as a training loss that stubbornly refuses to drop. That is the signal to raise the rank (giving the adapter more capacity) or to question whether SFT is the right method at all. But for the overwhelming majority of application tasks — format, style, classification, extraction, task distillation — low rank is not a compromise. It's a match to how little actually needs to change.

There's a second parameter that rides along with rank: alpha, a scaling factor that controls how strongly the adapter's output is added to the frozen base. A common convention sets alpha to roughly twice the rank, but the two interact — think of rank as the adapter's capacity and alpha as its volume knob. Turn the volume too high and a small adapter can still destabilize the base model; too low and even a high-capacity adapter barely moves behavior. When people say "tune LoRA," they mostly mean finding a sane rank/alpha pair and then leaving them alone.

The VRAM and compute budget: why QLoRA fits on one GPU

The single reason fine-tuning migrated from "cluster of expensive GPUs" to "one rented GPU over a weekend" is memory arithmetic. Understanding it demystifies why QLoRA is such a big deal, and lets you predict whether your run will fit before you pay for it.

When you train a model, GPU memory holds four things, not one:

  1. The model weights — the parameters themselves.
  2. The gradients — one number per trainable parameter, computed each step.
  3. The optimizer state — the modern default optimizer (Adam and its variants) keeps two extra numbers per trainable parameter (a running mean and variance). This is the sneaky one.
  4. Activations — intermediate values cached during the forward pass so the backward pass can use them; these scale with batch size and sequence length.

In full fine-tuning, items 2 and 3 apply to every weight. A rough rule of thumb for full fine-tuning in 16-bit is that you need on the order of ~16 bytes per parameter once weights, gradients, and optimizer state are counted — so a model with several billion parameters needs tens of gigabytes just for the training bookkeeping, before activations. That is why full fine-tuning wants multiple high-memory data-center GPUs.

LoRA changes the picture dramatically because gradients and optimizer state (items 2 and 3, the expensive ones) apply only to the adapter parameters — the well-under-1% you're actually training. The frozen base still occupies memory as weights, but it carries no gradients and no optimizer state. That alone can cut the training memory footprint by a large factor.

QLoRA attacks the remaining big cost — the frozen weights themselves — by loading the base model in 4-bit precision instead of 16-bit. That's roughly a 4× reduction on the largest remaining item. Combine the two moves — 4-bit frozen base (small weights) plus LoRA (no optimizer state on the base) — and a model that would have needed a small cluster to fine-tune now fits in the memory of a single GPU. The clever bit that makes 4-bit training viable rather than lossy is that the adapters are still trained in higher precision, so the gradient math stays stable even though the frozen base is compressed; the compression only ever affects the read-only part.

The costs you accept for this are modest and worth naming honestly: quantizing and dequantizing the base on the fly makes each training step somewhat slower, and 4-bit representation of the base introduces a small quality floor versus a full-precision fine-tune. For the vast majority of application tasks that floor is invisible in evaluation, which is exactly why QLoRA became the default. If you want the deeper story on what 4-bit actually discards and why inference-shaped work tolerates it, the quantization trade-offs post is the companion read.

Two practical planning notes. First, activations (item 4) are the variable you control at run time: if you run out of memory, cutting batch size or sequence length is the first lever, and gradient accumulation lets you keep an effective large batch while holding few examples in memory at once. Second, memory is not the only budget — wall-clock time and, if you're renting, dollars per GPU-hour matter too. A small clean dataset trained for a few epochs with QLoRA is often a single-digit-hours job on one GPU, which is precisely the economics that makes iterating on data (rather than agonizing over a giant one-shot run) the right strategy.

Hyperparameters that matter: LR, epochs, rank, alpha

The three knobs you'll actually touch: rank (adapter capacity — higher fits more complex behavior but risks overfitting; start low), learning rate (too high and you get instability or forgetting; fine-tuning uses smaller rates than you'd expect), and epochs (how many passes over the data — often just 1–3; more is usually overfitting, not learning). Change one thing at a time.

That's the summary; here's the reasoning, because with fine-tuning the difference between a working run and a wasted one is usually one of these being wrong.

Learning rate is the size of each corrective step. Set it too high and training becomes unstable — loss spikes, and the model lurches away from its pretrained competence (a fast route to catastrophic forgetting). Set it too low and the model barely moves; you burn compute and the fine-tune has no effect. Fine-tuning uses much smaller learning rates than pretraining, because you're adjusting an already-good model, not building one. A learning-rate schedule — a brief warm-up followed by a gradual decay — is standard and helps stability; the warm-up avoids a violent first step into a delicate set of weights. If you change one hyperparameter with intent, make it this one, and move it by factors (halve or double), not by tiny nudges.

Epochs — passes over your dataset — are where over-eagerness does the most damage. It is tempting to think "more training is more learning." Past a point it's the opposite: the model stops generalizing and starts memorizing your exact examples, which looks like beautiful training loss and disappointing held-out performance. For a clean, well-sized SFT dataset, 1 to 3 epochs is a normal range, and 1 is often enough. If your data is small, more epochs overfit faster; if it's large, you may not even need a full pass. Watch the gap between training loss and held-out loss: when training loss keeps dropping but held-out loss flattens or rises, you have already trained too long — the crossover is the overfitting cliff, and every step past it makes the model worse in the ways that matter while looking better on the metric that doesn't.

Rank and alpha we covered in the math section: rank is adapter capacity, alpha its scaling. Start at a low rank (8 or 16) and only raise it if the model clearly can't fit the behavior — i.e., training loss won't come down even with a sane learning rate. Raising rank to "be safe" mostly buys you a larger adapter and a faster path to overfitting, not better results.

Batch size interacts with learning rate and mostly affects stability and speed rather than final quality; if memory forces a small batch, use gradient accumulation to simulate a larger effective one. The discipline that ties all of this together is the same as any experiment: change one variable at a time, keep the evaluation fixed, and write down what each run did. Fine-tuning has few enough knobs that a methodical sweep of learning rate and epochs, holding data constant, will find a good setting quickly — and an undisciplined flail across all knobs at once will not, no matter how much GPU you throw at it.

The dataset is the whole game

You can get every hyperparameter right and still fail if the data is bad. You can get the hyperparameters mediocre and still succeed if the data is excellent. Spend your time here.

Format. Most instruction fine-tuning uses examples shaped like a conversation: an input (and any system framing) paired with the exact target output you want. The model learns to produce that target given that input. Whatever your serving format will be, your training examples must match it — same system prompt, same structure, same delimiters. Train/serve mismatch is one of the most common silent failures: the model performs beautifully in eval and mysteriously worse in production because the live prompt doesn't look like what it trained on.

The instruction-template trap. This deserves its own warning because it burns hours and produces symptoms that look like everything except the real cause. Instruction-tuned models were themselves trained with a specific chat template — the exact special tokens and delimiters that mark where the system prompt, the user turn, and the assistant turn begin and end. That template is part of the model's expected input; it is not decorative. When you build your fine-tuning dataset, your examples must be wrapped in the same chat template the model expects, and — critically — you must usually train on the completion only, masking the loss on the prompt tokens so the model is graded on producing the answer, not on parroting the question. Get the template subtly wrong (a missing turn token, the wrong role marker, an extra newline the model never saw in its own training) and you are teaching the model on inputs shaped unlike anything it will ever receive at inference. The result is a fine-tune that trained "successfully" — the loss went down — but underperforms in confusing ways, because it optimized for a format that doesn't exist in production. Most modern tooling applies the correct template for you if you let it; the trap is hand-rolling your own delimiters and quietly diverging from the one the model was built around. When a fine-tune is mysteriously worse than the base model, a template mismatch is the first thing to check.

Quality over quantity. A few hundred to a few thousand clean, consistent examples routinely beat tens of thousands of scraped, contradictory ones. "Consistent" is the operative word: if two near-identical inputs have differently-formatted outputs in your data, you're teaching the model to be inconsistent. Every labeling ambiguity you leave in the dataset comes back as unreliability in the model.

Consistency of labels. If humans built your targets, spot-check them. Decide edge cases up front and apply the rule uniformly. The model will faithfully learn your contradictions.

Cover the real distribution. Include the hard cases, the edge cases, and the "user typed something weird" cases — not just the clean happy path. A model fine-tuned only on tidy inputs falls apart on messy real ones.

Synthetic data, carefully. Generating training examples with a stronger model is a legitimate and popular shortcut, especially for bootstrapping. But it inherits the generator's blind spots and biases, so review it rather than trusting it wholesale. The synthetic data and distillation post covers doing this well — including using a big model to create data that teaches a small, cheap model one task competently, which is one of fine-tuning's genuinely great uses.

Hold out a test set now. Before you train anything, carve off examples the model will never see in training. This is your only honest measure of whether it worked.

Evaluate like you mean it

"It looks better" is not a result. Build the evaluation before you train, so success is defined in advance and you can't fool yourself after the fact.

  • Task metrics where they exist. Classification has accuracy and per-class breakdowns; extraction has exact-match and field-level scores. Use them.
  • A fixed held-out set. Run base model and fine-tuned model on the same held-out inputs and compare directly. If you can't beat a well-prompted base model, you haven't earned the fine-tune.
  • Regression checks. Test general behavior too, not just your target task. A model that got great at your format but now fails basic requests has forgotten more than it learned — you need to see that.
  • Human review on a sample. For open-ended outputs where metrics are weak, read a sample blind. It catches the "technically scored fine but subtly wrong" failures.

Two comparisons matter most: fine-tuned model vs. the base model with your best prompt (did tuning actually add value over prompting harder?), and fine-tuned model vs. the previous fine-tune (did this run improve on the last one?). Skip the first comparison and you'll ship training runs that a five-line prompt would have beaten.

The failure modes that waste your budget

None of these are exotic. All of them are common, and all are cheaper to avoid than to debug.

  • Catastrophic forgetting. Train too hard on a narrow task and the model gets worse at everything else. Guard with lower learning rates, fewer epochs, LoRA (frozen base helps), and regression evals that would catch it.
  • Overfitting to style. The model memorizes your examples' surface patterns instead of the underlying behavior, then flops on inputs shaped even slightly differently. Symptoms: near-perfect training loss, disappointing held-out results. Fix with more diverse data, fewer epochs, lower rank.
  • Train/serve prompt mismatch. Covered above and worth repeating because it's so common: the production prompt must match training. A stray system-prompt difference can erase your gains.
  • Solving the wrong problem. The big one. You fine-tuned for knowledge that should have been retrieval, or for behavior a prompt already handled. This wastes the most money because everything "worked" — you just didn't need it.
  • No baseline. If you never measured the well-prompted base model, you can't know whether fine-tuning helped or whether you spent a week matching what a prompt already did.
  • Ignoring inference economics. A fine-tune that needs a bigger model or special serving can cost more per request than the prompt-plus-big-model approach it replaced. Do the inference cost math before committing, especially if the goal was to save money.

When NOT to fine-tune

The decision tree tells you where fine-tuning wins. It's worth stating the losing cases directly, because these are the situations where teams fine-tune anyway and regret it. If any of the following describes you, close the training notebook.

You haven't seriously tried prompting and few-shot yet. This is the most common mistake by a wide margin. A vague prompt that a fine-tune "fixes" was never a fine-tuning problem; it was a prompting problem wearing a GPU. Exhaust the cheap, instant, editable options first. You lose nothing by trying, and you'll often be done in an afternoon.

Your real problem is knowledge, and it changes. If the thing you want the model to "know" lives in documents that update — prices, policies, inventory, docs, anything with a version — fine-tuning bakes in a snapshot that is stale the moment it changes and can only be updated by retraining. That's retrieval's job. Fine-tuning facts is choosing the one tool that makes the facts hardest to keep current.

You can't yet measure success. If you don't have a held-out set and a metric, you cannot tell whether a fine-tune helped, and you will end up shipping runs on vibes. Build the evaluation first; if you can't define what "better" means concretely, you're not ready to spend a training budget chasing it.

You have very little data, or dirty data. A few dozen inconsistent examples won't teach a stable behavior — they'll teach noise. If you can't assemble at least a few hundred clean, consistent examples that cover your real distribution, spend that effort on few-shot prompting instead, where a handful of examples is exactly the right amount.

The task needs more raw capability, not more specialization. If the model is failing because the problem is genuinely hard — novel reasoning, long chains of logic — fine-tuning on your examples won't make it smarter; it'll make it better at imitating your examples' surface. You want a stronger base model. Trying to fine-tune capability into a model that lacks it is the single most expensive way to be disappointed.

The economics don't close. If the fine-tune requires special serving, a bigger model, or infrastructure that costs more per request than the prompt-plus-big-model approach it replaces, you've spent effort to make things more expensive. Especially when "save money" was the goal, run the inference cost math first.

It's a one-off or low-volume task. Fine-tuning is an investment amortized over many requests. If you'll run the task a few hundred times, the training and maintenance cost dwarfs any per-request saving. Prompt it and move on.

The through-line: fine-tuning is worth it when the behavior is stable, demonstrable, and high-volume, and the cheaper rungs genuinely can't reach it. Absent those conditions, the most senior move is not to train — and to be able to explain, in one sentence, which cheaper tool solves the problem instead.

A minimal workflow

If you've walked the decision tree and fine-tuning is genuinely the answer, the path is short:

  1. Define success and build the eval set — held-out examples plus a metric — before touching training.
  2. Establish the baseline. Best-prompted base model on the eval set. This is the bar to beat.
  3. Assemble a small, clean, consistent dataset in your exact serving format. Prioritize quality and coverage over volume.
  4. Start with QLoRA, low rank, small learning rate, 1–3 epochs. Cheapest thing that could work.
  5. Evaluate against the baseline. Better on the task and no regression on general behavior? Good. Otherwise iterate on data first, hyperparameters second.
  6. Ship the adapter, keep the base fixed, and keep the eval set as a permanent regression guard for future versions.

To make that skeleton concrete, here is the same workflow walked as an end-to-end recipe for a realistic task — say, classifying inbound support tickets into your internal taxonomy and emitting a fixed JSON shape.

  1. Confirm it's a fine-tuning problem. The behavior (emit our taxonomy in our JSON) is stable, demonstrable, and high-volume, and a big system prompt full of the taxonomy is being paid for on every one of thousands of daily requests. Knowledge isn't the issue; format reliability and token cost are. Fine-tuning is on the table. (If the taxonomy changed weekly, you'd lean on retrieval or prompting instead.)
  2. Build the eval set first. Pull, say, 150–300 real tickets the model will never train on, and hand-label each with the correct category and target JSON. This is your held-out set. Define the metric up front: exact-match on the category, plus valid-JSON rate and field-level accuracy. Write it down so "better" is not negotiable after the fact.
  3. Establish the baseline. Run your best-prompted base model — a genuinely good prompt with a few in-context examples — against that eval set. Record the numbers. This is the bar. If it's already good enough, stop here; you don't need a fine-tune.
  4. Assemble the training data. Collect a few hundred to a few thousand tickets, labeled with the same rules as the eval set, formatted in the exact serving shape — same system framing, same chat template, same JSON schema — with loss masked to the completion. Resolve edge cases with a written rule and apply it uniformly. Include the messy, ambiguous, and adversarial tickets, not just clean ones. Keep the eval set strictly separate.
  5. Run the cheapest thing that could work. QLoRA on an instruction-tuned base, low rank (8–16), a small learning rate with brief warm-up, 1–3 epochs. On one GPU this is typically a few hours. Watch training vs held-out loss for the overfitting crossover.
  6. Evaluate against the baseline. Same eval set, same metric. Did the fine-tune beat well-prompted base on category accuracy and JSON validity? Run a regression check — does it still handle general requests and off-distribution tickets sanely, or did it forget? Read a blind sample of outputs to catch subtle wrongness the metric misses.
  7. Iterate on data first, hyperparameters second. If results disappoint, the culprit is usually the dataset — inconsistent labels, missing edge cases, or a template mismatch — far more often than the learning rate. Fix data, retrain, re-measure.
  8. Ship the adapter, freeze the base, keep the eval set forever. Serve the small adapter over the shared base, drop the giant taxonomy out of your per-request prompt (that's the cost win), and keep the eval set as a permanent regression guard so the next version has to prove it's an improvement.

Notice how much of this is not training. That's the point. The fine-tune is a small step surrounded by data work and measurement — and that surrounding work is what actually determines whether it succeeds.

FAQ

Can fine-tuning teach a model new facts about my company? Not reliably, and not in a way you can maintain. Fine-tuning adjusts behavior; facts shown a handful of times don't consistently stick, and any that do can't be updated without retraining. For knowledge — docs, policies, current data — use retrieval (RAG), which keeps facts current and lets you cite the source. Fine-tune for how the model responds, retrieve for what it knows.

How much data do I need to fine-tune an LLM? Less than most people expect, if the data is clean. A few hundred to a few thousand high-quality, consistent examples covering your real input distribution often outperform tens of thousands of noisy ones. Beyond a point, adding more contradictory or redundant data hurts. Fix quality and consistency before you chase volume.

What's the difference between LoRA and QLoRA? LoRA freezes the original model and trains small adapter matrices, so you update well under 1% of parameters and produce a tiny, swappable adapter file. QLoRA adds quantization — it loads the frozen base in a compressed 4-bit form to cut memory, letting you fine-tune large models on a single GPU. QLoRA trades a little quality and speed for a lot of memory savings; for most tasks it's the practical default.

Should I fine-tune a base model or an instruction-tuned model? Usually the instruction-tuned one. It already follows instructions and holds a conversation, so you're nudging existing behavior rather than teaching it from scratch — which needs far less data. Start from a base model only when you want to fully reshape behavior and have the data and expertise for it, which most application teams don't.

How do I know if my fine-tune actually worked? Compare it, on a held-out test set built before training, against the base model using your best prompt. If it doesn't clearly beat well-prompted base on your task and hold its own on general behavior (no catastrophic forgetting), it didn't work — regardless of how good the outputs feel. Define the metric in advance so you can't rationalize afterward.

Is fine-tuning cheaper than just using a bigger model with a good prompt? Sometimes, at high volume — a fine-tuned small model can match a big model on one narrow task at a fraction of the per-request cost, and shorter prompts save tokens. But it adds training and serving complexity, and if you got it wrong you pay for both. Run the per-request math and compare against the well-prompted big-model baseline before assuming fine-tuning saves money.

What's the difference between SFT and DPO, and do I need both? Supervised fine-tuning (SFT) teaches the model to imitate ideal answers from input/output pairs; it's the default and where you start. DPO (Direct Preference Optimization) is preference tuning — it learns from pairs of better vs worse answers to polish subtle quality like helpfulness or tone. Most application teams need only SFT. Reach for DPO as a second layer, after SFT, when the remaining problem is "which of two answers is better" rather than "what is the right answer" — and when you can supply preference pairs to express that. It's a refinement, not a starting point.

Can I fine-tune a hosted/closed model, or only open-weights ones? Both are possible but different. Several hosted providers offer managed fine-tuning: you upload a dataset, they train and serve an adapter behind their API, and you never touch the weights. It's convenient but less transparent and less portable. Open-weights models give you full control — LoRA/QLoRA on your own or rented GPU, an adapter file you own and can move — at the cost of running the pipeline yourself. If you want portability, cost control, or the ability to inspect and self-host, open weights win; the open-weights guide covers choosing one.

How long and how much does a fine-tune take? For the common case — QLoRA on an instruction-tuned model with a few hundred to a few thousand clean examples, low rank, 1–3 epochs — the training run itself is often a single-digit-hours job on one GPU, so the compute is cheap. The expensive part isn't the training; it's the data curation and evaluation around it, which can take days to weeks of human effort. That ratio is the whole lesson: budget your time for the dataset and the eval, not the training loop.

The bottom line

Fine-tuning is a precise instrument for a specific job: making a model reliably behave a way that's easy to demonstrate and hard to prompt. It is not a knowledge injector, not a general intelligence upgrade, and not a default first move. Walk the ladder — prompt, few-shot, retrieve, then tune — and most of the time you'll solve the problem before the last rung and save yourself a training pipeline you'd have to maintain forever.

When you do reach that last rung, remember where the work actually lives. The training run is the easy, automated part. The dataset and the evaluation are where fine-tuning is won or lost. Get those right, start with QLoRA and a small clean dataset, always measure against a well-prompted baseline, and you'll spend your GPU budget on the rare cases that genuinely need it — which is the whole point.