How to Choose an LLM for Your App: A Decision Framework

The honest answer to "which LLM should I use for my app?" is that you can't know until you've run a handful of candidates against your actual task, with your actual prompts, on your actual data. Every other answer — the leaderboard ranking, the vendor benchmark, the Twitter thread declaring a new king — is a proxy that may or may not correlate with the one thing you care about: does this model do my job well enough, fast enough, cheaply enough, and privately enough to ship?

So the framework below is not "use model X." It's a process. You define what "good enough" means for your use case, you write a small eval that measures it, you run three or four plausible candidates through that eval, and you pick the cheapest one that clears the bar. Then you re-run the eval every few months, because the model that wins today will be beaten — usually by a cheaper version of itself. If you take nothing else from this post: choosing a model is an ongoing measurement problem, not a one-time purchasing decision. This is a builder's guide; if you just want to know which chat product to use day to day, that's a different question answered in which AI chatbot.

Key takeaways
Step 1: Write down your constraints
The axes that actually matter
Step 2: Build a small eval on your real task
Why public benchmarks mislead
Step 3: Ladder down from the strongest model
Model routing and cascades
Step 4: Cost the decision properly
The cost math, worked against your traffic
Step 5: Open-weights vs closed API
Hosted API vs local and self-hosted
Step 6: Account for what actually moves accuracy
Step 7: Build for switching
Avoiding lock-in: the abstraction layer in practice
A worked example: choosing a model for support triage
Common mistakes
FAQ
The one-paragraph version

Key takeaways

Evaluate on your own task, not leaderboards. Public benchmarks measure general capability on someone else's problems. Build a 20–100 example eval set from your real inputs and score candidates on that.
There is no single "best" model — there's a best model for a given constraint set: capability, cost, latency, context length, privacy, and reliability. Rank your constraints before you shop.
Start with the strongest model to prove the task is possible, then move down to the cheapest model that still passes your eval. Capability buys you a working prototype; optimization buys you margins.
Cost is per outcome, not per token. A model that's 3x cheaper per token but needs two retries and a bigger prompt can cost more per resolved task.
Open-weights vs. API is mostly a privacy, control, and scale decision, not a quality one. Pick closed for speed-to-market, open when data residency, unit economics at high volume, or customization dominate.
Assume you'll switch. Build a thin abstraction over the provider so swapping models is a config change, not a rewrite.

Step 1: Write down your constraints before you look at any model

Most bad model choices come from optimizing one dimension while ignoring the others. Before comparing anything, force yourself to rank six constraints for this specific feature:

Capability — how hard is the task? Extracting a date from an email is easy; multi-step legal reasoning is not.
Cost — what can you spend per request and still have a viable product?
Latency — does a human wait on the output (chat, autocomplete) or does it run in a batch job overnight?
Context length — how much do you need to stuff into the prompt? A whole codebase and a one-line question have very different needs.
Privacy and data residency — can this data leave your infrastructure at all? Is there a regulatory boundary?
Reliability and format — do you need strictly valid structured output every time, or is prose fine?

The trick is that these trade against each other. The most capable model is usually the slowest and most expensive. The fastest is often the least capable. There is no model that wins all six, so you're really deciding which two or three you refuse to compromise on. A fraud-review pipeline might rank privacy and reliability first and happily accept high latency. A coding autocomplete feature ranks latency first and will accept a dumber model to get sub-200ms responses. Write these down. They are your scoring rubric.

A useful discipline here is to distinguish hard constraints from soft preferences. A hard constraint is a line you will not cross no matter how good a model is on every other axis: "patient data cannot leave our VPC," "p95 latency must be under 800ms or the feature is unusable," "the license must permit commercial use." Hard constraints eliminate candidates before you evaluate anything — they shrink the universe of models you even bother to test. Soft preferences are the axes you optimize within the surviving set: cheaper is better, faster is better, but you'll trade a little of each for enough of the others. Confusing the two is the most common planning error. Teams treat "cheap" as a hard constraint and eliminate the only model that actually solves the task, or treat "must be the smartest model" as a hard constraint and overpay by 10x for capability the task never uses.

Write the constraints as testable predicates, not adjectives. "Fast" is not a constraint; "median response under 1.2 seconds for a 500-token completion at our prompt length" is. "Cheap" is not a constraint; "under $0.004 per resolved ticket at 50,000 tickets a month" is. The moment a constraint is a number, it becomes something your eval can measure and something a candidate can pass or fail. Vague constraints produce vague decisions that get re-litigated every time someone reads a new benchmark thread.

One more framing that saves a lot of arguing: separate request-time constraints (what has to be true for a single call — latency, format, context) from program-level constraints (what has to be true for the system as a whole — total monthly cost, data residency, vendor risk, the ability to migrate). Request-time constraints are usually discovered by your eval. Program-level constraints are usually decided by your organization and are the ones that quietly veto the "obvious" technical winner. A model can pass every request-time check and still be disqualified because procurement won't sign the data-processing agreement.

The axes that actually matter

The six constraints in Step 1 are the ones most features rank on, but a rigorous shortlist scores candidates across a wider set of axes. Not every axis matters for every app — the point is to consciously decide which ones are load-bearing for yours rather than discovering a missing capability in production. Here is the full list, with the question each one answers and the trap each one hides.

Capability / quality. Can the model actually do the reasoning the task requires? This is the axis benchmarks approximate and the one your eval must confirm. The trap: "quality" is task-specific. A model that tops coding evals can be mediocre at faithful summarization, and vice versa. There is no scalar "smartness."
Latency. Two numbers matter, not one: time-to-first-token (what governs the feel of a streaming chat UI) and total generation time (what governs a batch job or a tool-calling loop). Reasoning models that "think" before answering can have excellent final answers and terrible time-to-first-token. Measure the number your UX actually depends on. Also watch tail latency (p95/p99), not just the median — one slow call in twenty is what users remember.
Cost. Input and output tokens are usually priced differently, often with output several times more expensive. Reasoning tokens, if the model emits them, bill as output. Batch and cached pricing can change the number by an order of magnitude. The trap is comparing sticker prices without your actual input:output ratio — a summarizer (huge input, tiny output) and a code generator (small input, huge output) have completely different effective costs from the same price sheet.
Context length. The advertised maximum is a ceiling, not a promise of quality. Many models degrade well before their stated limit — the "lost in the middle" effect, where information buried in the center of a long context is retrieved less reliably than information at the ends. If your task genuinely needs long context, put a long-context retrieval case in your eval rather than trusting the spec sheet. Larger contexts also cost more per call and add latency.
Tool-use / function-calling reliability. For anything agentic, the question is not "can it call tools" but "how often does it call the right tool with well-formed arguments, and does it know when to stop." This varies enormously between models that look equivalent on chat benchmarks. If your app is a tool-using agent, this axis can dominate all the others — a smarter model that hallucinates tool arguments is useless in a loop.
Structured-output support. Does the model support constrained decoding against a JSON schema, or are you parsing free text and praying? Native schema-constrained modes turn "usually valid JSON" into "always valid JSON," which is the difference between a model you can build a pipeline on and one you can't. A cheaper model with reliable structured output often beats a smarter model without it.
Modality. Text-only, or does the task need vision (reading screenshots, documents, charts), audio, or image output? Multimodal capability narrows the field sharply and is a hard constraint when present — no amount of text quality substitutes for the ability to read an image.
Privacy and hosting. Where does the data go, and what does the provider retain or train on? This spans from "zero-retention API endpoint" through "self-hosted in our own cloud" to "air-gapped on-prem." It is almost always a hard constraint set by legal or compliance, not an engineering preference.
Rate limits and throughput. Can the provider actually serve your peak traffic? A model you can't get enough tokens-per-minute of is not a model you can ship. New or in-demand models often have tight limits early. For self-hosted models, this becomes a capacity-planning problem you own.
Fine-tunability. Can you adapt the model to your task — full fine-tuning, LoRA adapters, or at least reliable few-shot steering? This matters when prompting plateaus below your accuracy bar. Open-weights models give you the whole toolbox; closed models offer a narrower, provider-controlled menu.
Licensing. For open-weights models especially, read the actual license. "Open" spans genuinely permissive (Apache-2.0, MIT) through source-available licenses with commercial-use thresholds or acceptable-use restrictions. This is a legal hard constraint that engineers routinely discover too late. The open-weights guide covers the license landscape in detail.
Stability and vendor risk. How likely is this model to change or disappear underneath you? Closed models get silently updated and eventually deprecated; a small provider might not exist next year. This axis rarely eliminates a candidate but strongly favors designing for switching (see Step 7).

The practical use of this list is not to score all twelve axes for every candidate — that's paralysis. It's to run down the list once per feature and mark each axis as hard constraint, load-bearing preference, or don't care. Most features have two or three load-bearing axes and treat the rest as pass/fail gates or ignore them entirely. The discipline is in doing this consciously, on paper, before a benchmark thread or a vendor demo anchors you on the wrong axis.

Step 2: Build a small eval on your real task

This is the step everyone skips and the step that actually decides the outcome. A leaderboard tells you how a model does on graduate physics questions or competition math. It tells you almost nothing about how it handles your customer-support tickets in your tone with your product names.

You do not need a research-grade evaluation harness. You need 20 to 100 representative examples from your real workload, each paired with either a known-good answer or a clear pass/fail rule. Concretely:

Collect real inputs. Pull actual queries, documents, or tickets — including the messy, ambiguous, and adversarial ones. The edge cases are where models differ most.
Define "correct." For extraction or classification, that's an exact-match or field-level check. For open-ended generation, write a rubric ("cites only provided sources," "refuses when data is missing," "stays under 100 words") and either grade by hand or use a stronger model as a judge.
Run every candidate through the same prompt. Keep the prompt fixed across models so you're measuring the model, not your prompt engineering. (Then, separately, tune the prompt for the winner — see how to write better prompts.)
Record accuracy, cost per run, and latency for each. Now you have a table you can actually reason about instead of vibes.

Fifty examples sounds small. It is enough to reveal a model that's systematically wrong on your task, which is the failure you most need to catch. You can grow the set over time; the first version's job is just to stop you from shipping on a hunch.

A few refinements separate an eval that survives contact with production from one that gives you false confidence. Stratify your examples so the set mirrors your real traffic mix — if 70% of your tickets are simple password resets and 30% are gnarly billing disputes, a set that's half hard cases will make every model look worse than it will perform live, and may push you toward an expensive model you don't need. Conversely, over-sample the tail on a separate slice so you can see how models handle the 5% of inputs that break things; report the two slices separately rather than averaging them into a single misleading number. A model that scores 92% overall but 40% on the adversarial slice is a different risk than one that scores 88% evenly.

Decide your grading method deliberately, because it determines what you can trust. Exact-match and field-level checks are cheap, deterministic, and unarguable — use them wherever the task has a right answer (extraction, classification, routing). Rubric grading by a stronger model (LLM-as-judge) scales to open-ended generation but introduces its own biases: judges favor longer answers, prefer their own family's style, and can be gamed. If you lean on a model judge, calibrate it against a few dozen human-graded examples first and keep a human in the loop on disagreements — the full method and its failure modes are covered in LLM-as-a-judge evaluation. The worst option is grading by vibes, where you skim a few outputs, decide one "feels smarter," and ship — that is exactly the hunch the eval exists to replace.

Account for non-determinism. The same model at the same prompt returns different outputs run to run, especially at higher temperature. Run each example two or three times and look at the spread, not a single sample. A model that's right on average but wildly inconsistent is a reliability problem your single-run eval would hide. Where the task allows it, pin temperature to zero for the eval so you're measuring the model's best-effort behavior rather than sampling noise — then test your production temperature separately if it differs.

Finally, version and store your eval like code. Commit the example set, the prompts, the grading rubric, and the results table to your repo. When a new model ships, re-running is a single command and the diff against last quarter's numbers is right there. An eval that lives in someone's notebook and gets rebuilt from memory each time is an eval you won't actually run, which means in practice you don't have one.

Why public benchmarks mislead

Public benchmarks are not useless — they're a legitimate way to build your initial shortlist and to sanity-check that a model is in the right weight class. The mistake is treating a leaderboard rank as a decision instead of a filter. Several structural problems make public numbers a poor final arbiter for your app.

Contamination. Popular benchmarks leak into training data. When test questions (or close paraphrases) appear in the pretraining corpus, a model can score well by partial memorization rather than capability. You can't audit this from the outside, and it inflates exactly the well-known benchmarks everyone quotes. Your private eval, built from your own recent inputs, is contamination-proof by construction.

Distribution mismatch. A benchmark measures a distribution of tasks that someone else chose. Graduate physics questions, competition math, and trivia say little about whether a model writes a faithful two-sentence summary of a support ticket in your product's voice, or reliably extracts an invoice total from a messy PDF. The correlation between "tops the leaderboard" and "best on my narrow task" is real but loose, and it gets looser the more specialized your task is.

Aggregation hides the shape. A single benchmark score is an average over many items. Two models with the same headline number can have completely different error profiles — one fails gracefully and rarely, the other is brilliant on average but catastrophically wrong on a category that happens to be 20% of your traffic. Averages are exactly the wrong summary for a decision where the tail is what hurts.

Metric gaming and teaching to the test. Once a benchmark matters commercially, it gets optimized against. Providers tune for the benchmarks that get quoted, which improves the number faster than it improves the underlying capability you care about. This isn't necessarily bad faith — it's Goodhart's law: a measure that becomes a target stops being a good measure.

Scoring on the wrong currency. Leaderboards rank on accuracy or win-rate. They never rank on cost per resolved task on your workload, which is the number that decides whether your feature has a viable margin. The leaderboard leader almost always loses on cost, often by a large multiple, to a model two rungs down that still clears your bar. That trade-off is invisible on every public ranking and central to your decision — see AI inference cost economics for how the per-resolution math actually works.

The healthy relationship with benchmarks: read them to assemble a shortlist of two to four plausible candidates and to notice genuinely new capabilities, then throw the rankings away and decide with your own eval. Use public numbers to find candidates; never use them to choose one.

Step 3: Ladder down from the strongest model

Once you have an eval, use a deliberate strategy rather than guessing. Start at the top of the ladder and climb down:

Prove feasibility with the most capable model available. If the best model on the market can't pass your eval, the task needs rethinking — better prompting, retrieval, decomposition into smaller steps, or tool use — not a different model. Don't optimize cost on a task that doesn't work yet.
Once it passes, walk down to cheaper and faster models. Try mid-tier and small models against the same eval. The question at each rung is binary: does it still clear the bar? The moment a model fails, stop; the one above it is your floor.
Consider a two-tier setup. Many production systems route easy cases to a small cheap model and escalate hard ones to a big model — either by a rule, a classifier, or a confidence check. This often beats picking a single middle model, because most real traffic is easy and a minority is hard.

This laddering is why capability and cost aren't really in tension over time: you use capability to discover that the task is solvable, then use the eval to buy back margin by dropping to the smallest model that still solves it.

Two subtleties make the ladder more useful than a naive "try each model" sweep. First, the failure mode tells you where to spend effort. If the strongest model passes and a mid-tier model fails, look at how it fails before writing it off. If it fails because of format or shallow prompting, better structured-output settings or a few examples might promote it back onto the ladder — you may not need the tier above at all. If it fails because it genuinely can't do the reasoning, that's a real capability floor and no amount of prompting rescues it. Distinguishing "failed for a fixable reason" from "failed for a fundamental reason" is where the ladder pays off, and it's why you re-run the eval after each prompt change rather than deciding once.

Second, the floor is a distribution, not a point. Because outputs are non-deterministic, a model that passes 89% and one that passes 91% may not be reliably distinguishable on a 50-example set — the gap is inside the noise. Don't treat a one- or two-point difference as decisive. Either grow the eval until the difference is stable, or (usually better) treat models within the noise band as tied and let the other axes — cost, latency, structured-output reliability — break the tie. The ladder's job is to find the cheapest tier that clears the bar, not to rank near-identical models to the decimal.

Model routing and cascades

The two-tier idea from Step 3 generalizes into one of the highest-leverage patterns in production LLM systems: routing and cascades. The insight is that real traffic is rarely uniform in difficulty. Most requests are easy and a minority are hard, so paying big-model prices on every request means overpaying on the easy majority. Routing sends each request to the cheapest model that can handle it specifically, rather than sizing the whole system for the hardest case.

There are two families, and they behave differently.

A cascade runs cheap-first and escalates on failure. Send every request to the small model. Check the result against a cheap signal — schema validation, a confidence score, a self-consistency check, or a lightweight verifier. If it passes, you're done at small-model cost. If it fails, escalate to the bigger model and pay the higher price only for that request. The economics are governed by two numbers: the fraction of traffic the small model handles (its "coverage") and the cost of the escalation check itself. If the small model resolves 80% of traffic and the check is cheap, your blended cost approaches the small model's price with the big model's reliability. The catch is double-billing on the hard 20% — those requests pay for both the small attempt and the big one, plus the check — so cascades win big when coverage is high and lose when the small model rarely succeeds.

A router decides up front which model to use. Instead of trying the cheap model and checking, a classifier (a rule, a small model, or a learned router) inspects the request and dispatches it directly to the right tier. This avoids the double-billing of a failed cheap attempt, but it moves the risk into the router: a misrouted hard request goes to the small model and produces a wrong answer with no escalation to catch it. Routers are better when the escalation check is expensive or unreliable; cascades are better when a cheap, trustworthy check exists.

Deciding between them comes down to whether you have a good, cheap verifier. Tasks with checkable outputs — valid JSON against a schema, code that compiles and passes tests, a math answer you can verify — are perfect for cascades, because the check is nearly free and nearly perfect. Tasks with no cheap correctness signal — open-ended generation, subjective quality — favor a router, because there's nothing reliable to escalate on.

Three warnings before you build one. Routing adds real complexity: another component to monitor, tune, and debug, and a new failure surface when the router itself is wrong. Measure the actual blended economics against a single mid-tier model on your traffic before committing — sometimes one well-chosen middle model beats a cascade once you price in the check and the engineering. And route on your eval, not your intuition: the whole system, router included, has to clear the same bar, so extend your eval to score the routed pipeline end to end rather than scoring each model in isolation.

Step 4: Cost the decision properly

Per-token pricing is the headline number and the most misleading one. What matters is cost per successfully completed task, which folds in things the sticker price hides:

Retries. A less reliable model that fails validation and re-runs 20% of the time is 20% more expensive than its token price implies — and adds latency.
Prompt overhead. A weaker model may need more few-shot examples, longer instructions, or more retrieved context to hit the same accuracy, inflating input tokens on every call.
Output verbosity. Reasoning-style models can emit far more tokens to reach an answer. Sometimes worth it; sometimes you're paying for thinking you didn't need.
Caching. If your prompts share a large fixed prefix (a system prompt, a document, tool definitions), prompt caching can cut input costs dramatically — but only some models and setups support it well.

The mental shift is from "cost per token" to "cost per resolution." A model that's twice the token price but needs no retries, a shorter prompt, and terser output can be the cheaper choice in practice. For the full breakdown of what actually drives the bill, see AI inference cost economics.

Two more factors sit outside the per-call arithmetic and routinely dominate it. The input:output ratio determines which price actually matters. Output tokens usually cost several times more than input tokens, so a task that reads a long document and emits a one-line label is priced almost entirely on cheap input, while a task that takes a short instruction and generates a long draft is priced almost entirely on expensive output. Two models with identical headline prices can have very different effective prices on your specific ratio. Never compare price sheets without multiplying by your own token profile. The human cost of errors is the factor that makes cheap models expensive in disguise: if a wrong answer triggers a support escalation, a manual correction, or a refund, the fully loaded cost of that error can dwarf thousands of correct inference calls. A model that's cheaper per token but wrong 3% more often on a task where errors are costly is not the cheaper model — it just moves the cost off your cloud bill and onto your operations team, where it's harder to see.

The clean way to think about it: your true unit cost is (inference cost per attempt × average attempts per resolution) + (error rate × cost per error). The first term is what dashboards show you. The second term is usually larger and almost never on the dashboard. A model-selection decision made only on the first term is a decision made on the smaller number.

The cost math, worked against your traffic

Abstract advice about "cost per resolution" gets concrete the moment you plug in your own traffic. The arithmetic is simple; the discipline is doing it with your real numbers instead of the sticker price. Here is the shape of the calculation, using placeholder rates so you can substitute your own — the method is what matters, not any specific price, which will be stale by the time you read this.

Start with your per-call token profile from the eval. Suppose a support-triage call averages 1,500 input tokens (the ticket plus a system prompt plus a couple of retrieved help-desk articles) and 200 output tokens (a category, a priority, and a one-line rationale). At a hypothetical rate of $3 per million input tokens and $12 per million output tokens, one call costs (1,500 × $3 + 200 × $12) / 1,000,000 = $0.0045 + $0.0024 = $0.0069, call it seven-tenths of a cent. That's the number a naive comparison stops at.

Now layer in the factors that actually decide it. Say this model passes your schema check 94% of the time and the 6% of failures get one automatic retry, and the retry succeeds. Average attempts per resolution is 1 + 0.06 = 1.06, so inference cost per resolved ticket is $0.0069 × 1.06 ≈ $0.0073. Now suppose 2% of tickets are still mis-triaged after the retry, and a mis-triage costs your team an average of $4 in manual rework and delayed response. The error term is 0.02 × $4 = $0.08 per ticket — more than ten times the inference cost. Your true unit cost is about $0.0073 + $0.08 ≈ $0.087 per resolved ticket, and it is dominated entirely by the error rate, not the token price.

That single observation reorders the whole decision. A model that costs twice as much per token but cuts the post-retry error rate from 2% to 0.5% changes the error term from $0.08 to $0.02, dropping true unit cost from ~$0.087 to ~$0.035 — cheaper overall despite the higher sticker price, and better for users besides. This is the concrete version of "cost per resolution beats cost per token," and you can only see it by writing the arithmetic down with your own numbers.

Scale it to your volume to size the decision. At 50,000 tickets a month the naive token cost is about $345, which is small enough that engineers often stop caring — but the fully loaded cost is closer to $4,350, and the difference between the two models above is roughly $2,600 a month, or $31,000 a year, before counting the user-experience gain. At a million tickets a month those numbers get an order of magnitude larger and the model choice becomes a material line item. The volume tells you how much the decision is worth, and therefore how much effort the eval deserves. For a feature serving a hundred calls a day, spend an afternoon and move on. For one serving millions, the ladder and the cost math earn their keep many times over.

Two mechanics to fold in when they apply. Prompt caching: if every call shares a large fixed prefix — a long system prompt, a stable set of tool definitions, a document you ask many questions about — caching that prefix can cut effective input cost dramatically, but only for models and setups that support it, and only when your traffic pattern actually reuses the prefix within the cache window. Batch pricing: work that doesn't need a real-time answer (overnight enrichment, bulk classification) often qualifies for steeply discounted batch endpoints. Both can change the ranking, so include them in the eval's cost column rather than bolting them on afterward.

Step 5: Decide open-weights vs. closed API

This is framed as a quality debate but is mostly a control and economics decision. Both categories have models capable of serious production work; the differences that matter are structural.

Dimension	Closed API	Open-weights (self-hosted)
Time to first working version	Fastest — a key and an HTTP call	Slower — you provision and run inference
Data path	Leaves your infra to a provider	Can stay entirely inside your boundary
Unit cost at low volume	Cheap — you pay per call	Expensive — you pay for idle GPUs
Unit cost at high, steady volume	Can get expensive	Can be far cheaper once utilization is high
Customization	Prompting and light fine-tuning	Full fine-tuning, quantization, control of the stack
Ops burden	Provider handles it	You own uptime, scaling, upgrades
Version stability	Provider may deprecate or silently update	You pin a version and keep it forever

Reach for a closed API when speed to market matters most, volume is modest or spiky, and your data can legally leave your systems. Reach for open-weights when data residency or privacy is non-negotiable, when volume is high and steady enough to keep GPUs busy, when you need deep customization, or when you want a model version that will never change underneath you. Plenty of teams do both — closed models to prototype and validate the task, then a migration to open-weights once the workload is proven and the economics or privacy case is clear. The full trade study lives in the open-weights guide.

The most common analytical error here is comparing an open-weights model's token price — often zero, since you're running your own weights — against a closed API's per-token bill and concluding open is free. It isn't. Self-hosting substitutes a token bill for a capacity bill: you pay for GPUs whether or not requests are flowing, plus the engineering time to run inference, autoscale, monitor, patch, and stay ahead of new model releases. The break-even is a utilization question. At low or spiky volume, idle GPUs make self-hosting far more expensive per resolved task than an API you only pay for when you call it. At high, steady volume that keeps accelerators busy, the fixed cost amortizes across enough requests that self-hosting can undercut the API by a wide margin. The crossover point is specific to your traffic shape, your hardware, and the model size — compute it, don't assume it. And weigh the hidden line item that never appears in either price: the ops burden. Running production inference is a real, ongoing engineering commitment. If that's not a muscle your team has or wants, the API's higher token price is often the cheaper total-cost-of-ownership once you count the salaries.

A subtler point: open vs closed is not actually a quality axis anymore. Both categories contain models fully capable of serious production work, and the gap that used to justify "closed is just smarter" has narrowed to the point where, for most application tasks, your eval will show a capable open model clearing the same bar as a closed one. So decide this on the structural axes — data path, unit economics at your volume, customization needs, version stability, ops appetite — not on a vague sense that one category is inherently better. Let the eval settle capability; let the constraints settle everything else.

Hosted API vs local and self-hosted

"Open vs closed" and "hosted vs local" are related but not the same axis, and conflating them causes muddled decisions. Open vs closed is about the weights — can you download and modify them. Hosted vs local is about where inference runs — someone else's infrastructure or yours. The two combine into a spectrum, and most of the interesting choices live in the middle.

Closed, hosted (the default API). You send requests to a provider's endpoint and pay per token. Fastest to ship, zero ops, but your data leaves your boundary and you're subject to the provider's rate limits, pricing, and deprecation schedule.
Open-weights, hosted by a third party. You run an open model, but on someone else's serving infrastructure (a managed inference provider). You get the open model's licensing freedom and version stability without owning GPUs, at the cost of the data still transiting a third party. A pragmatic middle ground when you want open-model economics and pinnability but not the ops.
Open-weights, self-hosted in your own cloud. The weights and the inference both live inside your infrastructure boundary. This is what data-residency and privacy constraints usually require, and where high-volume unit economics can win — but you own capacity planning, scaling, and uptime.
Local / on-device or on-prem. The model runs on hardware you physically control — a workstation, an on-prem server, or an air-gapped machine. This is the extreme end of the privacy spectrum: data never touches a network you don't own. It's also the most constrained on model size and throughput, since you're limited to what your local hardware can run. For prototyping, privacy-sensitive personal tooling, or genuinely offline requirements, running a model locally is increasingly practical — the run LLMs locally guide walks through what's realistic on consumer and workstation hardware.

The decision procedure is the same as everywhere else: your hard constraints usually collapse this spectrum to one or two viable points before capability enters the picture. A strict data-residency rule eliminates everything but self-hosted or local. A tiny team with spiky traffic and no privacy constraint has little reason to leave the hosted API. The middle options exist precisely for the teams whose constraints don't cleanly land at either extreme — who need open-model licensing or pinnability but can't justify running GPUs, or who need most of the privacy of self-hosting without the full ops load. Map your constraints onto the spectrum first; only then compare the surviving options on cost and capability.

Step 6: Account for what actually moves accuracy

Before you conclude that model A is smarter than model B, rule out the things that dominate model choice in real systems:

Retrieval quality. For any task grounded in your own data, what you put in the context window usually matters more than which model reads it. A mediocre model with excellent retrieval beats a great model fed the wrong documents. That's a RAG and embeddings problem, not a model-selection problem.
Prompt structure. Clear instructions, good examples, and a well-specified output format routinely close the gap between adjacent model tiers.
Structured output and validation. If you need JSON, use the model's structured-output or schema-constrained mode and validate every response. This often lets a smaller model be reliable enough to use.
Tool use and decomposition. Breaking a hard task into smaller, checkable steps — each a simpler model call — frequently beats throwing the whole thing at the biggest model in one shot.

The reason this matters for selection: if you fix these first, a cheaper model often clears your bar, and you never needed the expensive one. Optimizing the system around the model is usually a better return than optimizing the model.

There's an ordering implication worth stating plainly: do the system work before the model shootout, not after. If you evaluate models on a weak prompt with poor retrieval and no output validation, you're measuring how well each model compensates for a broken system — which rewards raw capability and pushes you toward the expensive model for the wrong reason. Fix the retrieval, tighten the prompt, add schema-constrained output, decompose the task into checkable steps, then run the ladder. Very often the cheaper model that failed on the naive setup passes comfortably on the improved one, and the "we need the biggest model" conclusion evaporates. The model is one component in a pipeline; selecting it in isolation from the pipeline that feeds it produces a decision that's both more expensive and less robust than it needs to be.

Step 7: Build for switching

The single safest assumption in this whole space is that your choice will be obsolete within a year — beaten by a cheaper, faster successor, often from the same provider. Design for that from day one:

Abstract the provider. Route model calls through one internal interface so swapping is a config change, not a code change. Avoid leaking provider-specific quirks throughout your codebase.
Keep the eval as a regression test. When a new model appears, run your existing eval set. You'll know in an hour whether it's an upgrade, and you'll catch regressions when a provider silently updates a model behind the same name.
Watch for deprecation. Closed providers retire model versions. Pin versions explicitly and have a migration path, so a deprecation notice is a scheduled task rather than an outage.
Re-shop on a cadence. Put a recurring reminder — quarterly is reasonable — to re-run the ladder. The cheapest model that passes your eval today is often not the cheapest one that passes it next quarter.

Treat model choice like a dependency you keep patched, not a foundation you pour once.

Avoiding lock-in: the abstraction layer in practice

"Abstract the provider" is easy advice to nod at and easy to implement badly. The goal is concrete: when a better or cheaper model arrives, or a provider raises prices, tightens rate limits, or deprecates a version, switching should be a scoped, testable change — not a search-and-replace across your codebase and a week of regression bugs. Here's what the abstraction actually needs to cover, and where teams get it wrong.

Centralize the call site. Every model invocation should go through one internal module — a complete() or generate() function your code calls, which internally decides which provider and model to hit. No feature code should import a provider SDK directly. This is the part everyone gets right at first and erodes over time, as someone reaches for a provider-specific feature "just this once." Guard it in code review; a single leaked SDK import is where lock-in creeps back.

Normalize the request and response shapes. Providers differ in how they represent messages, system prompts, tool definitions, token limits, stop sequences, and streaming. Your abstraction should expose one neutral shape and translate to each provider's dialect internally. The response side matters just as much: normalize token usage, finish reasons, and tool-call structures so downstream code never branches on which provider answered.

Don't over-abstract to the point of erasing capability. The failure at the other extreme is a lowest-common-denominator wrapper that only exposes features every provider shares, so you can never use a model's best capability — structured-output modes, prompt caching, provider-specific tool-calling — without going around your own abstraction. The right design lets provider-specific features through as optional, clearly-marked extensions, so using them is a deliberate choice you can find and cost when you migrate, not an accident scattered everywhere. Abstraction is about making switching cheap and visible, not about pretending all models are identical.

The eval is the other half of the abstraction. A thin interface lets you swap models mechanically; the eval tells you whether the swap is safe. Together they turn "should we try the new model" from a risky rewrite into a one-command experiment: point the interface at the new model, run the eval, read the diff. Without the eval, the abstraction just lets you break things faster. Keep both, and model migration becomes routine maintenance instead of a project.

A note on gateways and libraries: there are off-the-shelf abstraction layers and model gateways that give you a unified API across providers, plus routing, caching, and spend controls. They can save real work, but they're an architectural dependency of their own — evaluate one on the same axes you'd evaluate a model, and make sure it doesn't become the very lock-in you adopted it to avoid.

A worked example: choosing a model for support triage

Walking one decision end to end shows how the steps compose. The feature: automatically triage inbound support tickets — assign a category, set a priority, and draft a one-line internal note — for a mid-size SaaS product handling roughly 50,000 tickets a month.

Constraints (Step 1). We rank them: reliability of structured output is load-bearing (the category and priority feed a routing system, so malformed output breaks the pipeline); cost matters at this volume; latency is soft (triage runs in the background, a few seconds is fine); privacy is a hard constraint but satisfiable — tickets can go to a zero-retention API endpoint under our existing data-processing agreement, so both hosted and self-hosted stay on the table. Capability requirement is moderate: this is classification plus short generation, not deep reasoning.

Axes that matter (the axes section). Marking the full list: structured-output support — hard gate; cost — load-bearing; capability — pass/fail at a moderate bar; latency — don't care within reason; tool use, modality, long context, fine-tunability — don't care for this task. That leaves us optimizing cost among models that reliably emit schema-valid JSON and clear a moderate accuracy bar.

Eval (Step 2). We pull 80 real tickets, stratified to match live traffic: mostly routine (password resets, how-to questions), with a deliberate slice of hard cases (angry multi-issue tickets, billing disputes, non-English). Ground truth is the category and priority a senior support agent assigned. Grading is field-level exact match for category and priority; the drafted note we spot-check with a rubric. We run each ticket at temperature zero, twice, to check consistency.

Ladder (Step 3). We start with a top-tier model to prove the task is solvable: it scores 96% on category, 91% on priority, always valid JSON. Feasible. Then we climb down. A mid-tier model: 94% / 89%, still always valid JSON — inside the noise of the top model on our 80-example set, so effectively tied on accuracy at a fraction of the price. A small, cheap model: 90% category but only 78% priority, and it occasionally emits malformed JSON on the messy tickets — it fails the structured-output gate. So the mid-tier model is our floor.

Routing (the routing section). Could a cascade do better? The small model handles the routine 70% acceptably and only stumbles on the hard tail. We prototype a cascade: small model first, escalate to the mid-tier model when JSON validation fails or the model's own confidence is low. On our eval the cascade matches the mid-tier model's accuracy at roughly 60% of its cost. Worth it at 50,000 tickets a month; we'd have skipped it at 500.

Cost math (the cost-math section). We compute true unit cost including the ~2% residual mis-triage rate and the ~$4 rework cost of a bad triage. The error term dominates, so we confirm the mid-tier-in-the-cascade model — not the cheapest raw model — minimizes cost per resolved ticket, and we keep the top-tier model wired in as the escalation target for the hardest cases.

Hosting (Steps 5 and the hosted-vs-local section). At 50,000/month with a satisfiable privacy path via a zero-retention endpoint, self-hosting's idle-GPU cost isn't justified — utilization would be too low. We stay on the hosted API and revisit if volume grows an order of magnitude.

Switching (Step 7 and lock-in). All of this runs behind one triage() interface. The eval is committed to the repo. When a new model ships next quarter, we point the interface at it, run the 80-example eval, and read the diff — a one-afternoon experiment, not a rebuild.

The result isn't "use model X." It's a defensible decision with the receipts to re-make it cheaply when the landscape shifts — which is the only kind of model decision worth making.

Common mistakes

The same handful of errors account for most regretted model choices. Named plainly so you can catch them in yourself:

Choosing on the leaderboard. Picking the top-ranked model and shipping, without ever measuring it on your own task. The rank is a filter for building a shortlist, never a decision.
Evaluating by vibes. Skimming a few outputs, deciding one "feels smarter," and calling it evaluation. Without a scored eval on representative inputs, you're pattern-matching on a handful of cherry-picked examples.
Comparing token prices instead of resolution cost. Ranking models on the price sheet while ignoring retries, prompt overhead, output verbosity, your input:output ratio, and the human cost of errors — the factors that usually dominate the bill.
Over-provisioning capability. Paying for the biggest model on a task that a small model plus good prompting and validation would solve. Capability you don't use is margin you burn on every single call, forever.
Under-provisioning and blaming the model. The opposite error: concluding a task is impossible after testing one weak model on a bad prompt with no retrieval, when the strongest model would have proven it feasible and pointed at what to fix.
Ignoring the tail. Optimizing the average and getting blindsided by the 5% of adversarial inputs that produce the failures users actually notice and remember.
Treating the choice as permanent. Wiring one provider's SDK through the whole codebase, so the inevitable better-and-cheaper successor is a rewrite instead of a config change.
Skipping the system work. Running the model shootout before fixing retrieval, prompting, and output validation — which measures how well each model compensates for a broken pipeline and pushes you toward the expensive model for the wrong reason.
Confusing hard constraints with soft preferences. Eliminating the only model that solves the task because it's not the cheapest, or overpaying massively to satisfy a "must be smartest" preference that was never actually a requirement.

FAQ

Should I just use the top model on the leaderboard? Leaderboards measure general capability on standardized tasks that are probably not yours. Use them to build a short list of two to four plausible candidates, then decide with an eval built from your real inputs. The leaderboard leader frequently loses on your specific task to a cheaper model, and always loses on cost.

How big does my eval set need to be? Start with 20 to 100 real, representative examples — including the messy and adversarial ones, since that's where models diverge. That's enough to catch a model that's systematically wrong on your task, which is the mistake that hurts most. Grow the set over time and keep it as a regression test for future models.

Is a bigger model always more accurate? No. Bigger models are more capable on hard, open-ended reasoning, but for many production tasks — classification, extraction, routing, short structured responses — a small model with good prompting, retrieval, and validation matches them at a fraction of the cost and latency. Only pay for capability the task actually requires.

When should I self-host an open-weights model instead of calling an API? When data cannot leave your infrastructure, when your volume is high and steady enough to keep GPUs well-utilized, when you need deep customization or fine-tuning, or when you need a model version that will never change underneath you. For fast prototyping, spiky traffic, or low volume, a closed API is usually cheaper and far less operational work.

How do I actually compare cost between models? Measure cost per successfully completed task, not per token. Fold in retry rates from failed validations, the prompt length each model needs to hit your accuracy bar, output verbosity, and any prompt-caching savings. A model with a higher token price can be cheaper per resolved task once those factors are included.

How often should I revisit my model choice? Quarterly is a sensible default. Keep your eval set as a standing regression test and re-run the ladder — strongest model down to cheapest that still passes — whenever a notable new model ships or a provider announces a deprecation. The winner changes often, usually in your favor on price.

Should I use one model for my whole app or different models per feature? Almost always different models per feature. "Which LLM for my app" is the wrong unit of decision — the right unit is the individual task. A summarization feature, a code-generation feature, and a classification feature have different constraint rankings and will land on different models. Standardizing on one model across all of them means overpaying on the easy tasks to satisfy the hardest one. Run the framework per feature; share the abstraction layer and the eval harness across them, not the model.

Is a model router or cascade worth the added complexity? It depends on your volume and whether you have a cheap, reliable way to check a small model's output. At low volume, the engineering and monitoring overhead usually outweighs the savings — pick a single model that clears your bar. At high volume with a good verifier (schema validation, compiling code, a checkable answer), a cheap-first cascade can match a big model's reliability at a fraction of its cost and easily justify the complexity. Always measure the blended economics of the full routed pipeline against a single mid-tier model on your real traffic before committing; sometimes the simpler option wins.

How do I evaluate open-ended generation where there's no single correct answer? Write an explicit rubric — the specific properties a good output must have (grounded only in provided sources, correct length, right tone, refuses when data is missing) — and grade against it rather than against a gold answer. You can grade by hand for small sets or use a stronger model as a judge for scale, but calibrate the judge against human-graded examples first, because model judges have biases (favoring longer or same-family outputs) that will quietly skew your decision. The full method and its pitfalls are in LLM-as-a-judge evaluation.

The one-paragraph version

Rank your constraints, build a small eval from your real task, prove feasibility with the strongest model, then ladder down to the cheapest model that still passes. Cost the decision per resolved task, not per token. Choose closed APIs for speed and open-weights for control, privacy, and high-volume economics. Fix retrieval, prompting, and validation before blaming the model. And build a thin abstraction so that when — not if — a better, cheaper model arrives, switching is a config change and a re-run of your eval.

Table of contents