Prompt20
NewsDataBlog
All posts
local-llmollamallama-cppggufquantizationprivacyoffline-aiappliedevergreen

How to Run LLMs Locally: A Practical Guide to Private, Offline AI

Running open models on your own machine with tools like Ollama, LM Studio, and llama.cpp. GGUF and quantization sizing, VRAM vs RAM, what hardware you need, and when local genuinely beats the cloud.

By Prompt20 Editorial · 30 min read

You can run a genuinely useful large language model on a laptop you already own, entirely offline, for the cost of a download. That sentence would have been a lie a few years ago and is a mild understatement now. The catch is that "useful" is doing a lot of work: the local model you can run today is smaller and dumber than the frontier model behind a chat box, and pretending otherwise is how people end up disappointed. This guide is about setting expectations correctly and then hitting them.

The good news is that the hard parts — the concepts you actually need — barely change even as the tools and model names churn every few months. If you understand three things — how quantization trades size for quality, how to do the memory math that tells you what will fit, and which category of tool does what — you can pick up whatever this month's favorite is in an afternoon. That is what we'll cover. Names are snapshots; the mental model is the durable asset.

This guide goes deeper than the usual "install Ollama and run a command" walkthrough. We'll do the actual arithmetic — bytes per parameter at each quant level, the memory cost of a growing context window, what a KV cache is and why it eats your VRAM at long context lengths — and we'll be specific about the trade-offs the marketing glosses over. By the end you should be able to look at any model card and any spec sheet and predict, before downloading a single gigabyte, roughly how fast it will run and whether it will run at all.

Table of contents

Key takeaways

  • Local LLMs are real and private, but not frontier-class. A model that fits on consumer hardware trades raw capability for privacy, offline access, zero per-token cost, and full control. Know which of those you actually need before you start.
  • The one number that decides everything is memory. A model's file size (in its quantized form) plus some overhead has to fit in your VRAM, or in unified/system RAM. Do this arithmetic first; it saves hours.
  • Quantization is the core trick. It shrinks a model by storing its weights in fewer bits. A 4-bit quant is roughly a quarter the size of the full-precision model with a small, usually-acceptable quality hit. Everything below ~3-bit degrades fast.
  • GGUF is the format, and tools are categories, not brands. There's a one-line launcher tier (Ollama), a GUI tier (LM Studio), and an engine tier (llama.cpp, plus GPU-first servers). Learn the tiers; swap the brands freely.
  • Your GPU's VRAM (or a Mac's unified memory) is the real ceiling. CPU-only inference works but is slow. "Partial offload" lets you run models slightly bigger than your VRAM at a speed penalty.
  • Context length has its own memory cost. The KV cache grows linearly with how much text you feed the model and can rival the weights themselves at long contexts. A model that fits at 4K tokens may not fit at 32K.
  • Tokens per second, not benchmark scores, decides whether you keep using it. A "smart but 2 tok/s" model loses in practice to a "decent but instant" one, because you'll only reach for the fast one.
  • Local wins for privacy, offline use, high volume, and tinkering. Cloud wins for peak intelligence and zero setup. Most people end up using both.

What "running locally" actually means

Running an LLM locally means the model weights live on your machine and inference — the act of generating text — happens on your own CPU or GPU. Nothing leaves the device. No API key, no account, no network call. When you type a prompt, your hardware does the matrix multiplications and streams tokens back.

This is a different thing from the chat product you're used to. A hosted assistant is a giant model running on a cluster of expensive accelerators, wrapped in a product with search, memory, tools, and safety systems. What you run locally is just the raw model — the engine, not the car. If you want to understand what that engine is doing under the hood, our explainer on how AI chatbots work covers the tokens-in, tokens-out loop that's identical whether the model is on your desk or in a data center.

Two motivations drive people to local. The first is privacy: your prompts, your documents, your half-formed ideas never touch someone else's server. For journalists, lawyers, clinicians, and anyone handling sensitive data, that's not a nice-to-have. The second is control and cost: no rate limits, no per-token billing, no model getting silently swapped or deprecated underneath you, and it keeps working on a plane or in a blackout. Neither of those requires the model to be the smartest one in existence — just good enough for the job.

It helps to be precise about the mechanics, because "the model runs on my machine" hides several moving parts. At load time, the runner reads the weights file off disk and loads it into memory — VRAM if you have a GPU, system RAM otherwise. This is a one-time cost per session; a 5 GB model takes a few seconds to load from an SSD and then sits resident. Generation itself is memory-bandwidth-bound, not compute-bound, for single-user inference: producing each token requires reading essentially every weight in the model once. That single fact explains most of local LLM performance. It's why a GPU with 900 GB/s of memory bandwidth generates tokens far faster than a CPU with 50 GB/s even when both technically "fit" the model, and why the size of the model in bytes — not its parameter count in the abstract — predicts speed. A 4 GB model on a device with 100 GB/s of usable bandwidth caps out around 25 tokens per second in the best case (100 ÷ 4), before any overhead. Keep that back-of-envelope in mind; it turns "will it be fast enough?" into arithmetic.

There is also a distinction worth drawing between prompt processing (also called prefill) and token generation (decode). When you submit a prompt, the model first ingests all of it in one parallel pass — this is compute-heavy and fast, measured in the hundreds or thousands of tokens per second even on modest hardware. Then it generates the response one token at a time, each token depending on the last — this is the bandwidth-bound, slower phase. This is why feeding a model a long document and asking for a one-word answer feels quick, while asking for a long essay from a short prompt feels slow: the expensive part is the length of what it writes, not the length of what it reads.

The one calculation that matters: memory math

Before you download anything, learn to answer one question: will it fit? Almost every "why is this so slow" or "why did it crash" problem traces back to memory.

A model's memory footprint is dominated by its weights — the learned parameters. The rough size of the weights file is:

parameters × bits-per-parameter ÷ 8 = bytes

So a 7-billion-parameter model at full 16-bit precision is about 7e9 × 16 ÷ 8 ≈ 14 GB. Quantize it to 4 bits and it's about 7e9 × 4 ÷ 8 ≈ 3.5 GB. On top of the weights you need headroom for the context (the KV cache, which grows with how much text you feed in) and general overhead — budget roughly 1-3 GB extra for typical use, more for very long contexts.

The decisive question is where that memory lives:

  • VRAM (dedicated GPU memory) is fastest. If the whole model fits in VRAM, you get the best speed.
  • Unified memory (Apple Silicon Macs, some newer chips) is shared between CPU and GPU and is nearly as good — this is why Macs punch above their weight for local LLMs.
  • System RAM with CPU inference works and is cheap, but it's much slower, especially as the model grows.

Here's the sizing intuition as a table (approximate weight sizes at 4-bit quantization):

Model size ~4-bit weights Fits comfortably in Realistic experience
1-3B ~1-2 GB Almost anything, even phones Fast; fine for simple tasks, autocomplete, drafts
7-9B ~4-5 GB 8 GB VRAM or 16 GB RAM The sweet spot for most laptops
12-14B ~7-9 GB 12 GB VRAM or 16-24 GB unified Noticeably smarter; needs a real GPU or a Mac
27-34B ~16-20 GB 24 GB VRAM or 32 GB+ unified Strong; workstation or high-end Mac territory
70B+ ~40 GB+ 48 GB+ VRAM or 64-128 GB unified Serious hardware; approaches cloud-lite quality

The takeaway: match the model tier to your memory first, then pick a specific model within that tier. Downloading a 70B model onto a 16 GB laptop and wondering why it crawls is the most common beginner mistake.

One more subtlety the table smooths over: you cannot use 100% of your VRAM for the model. The operating system, your display, the browser you left open, and the runner's own working buffers all take a cut. On a GPU, budget losing 1-2 GB to overhead before the model even loads; on a shared-memory Mac, the OS reserves a chunk of unified memory that macOS will not hand to the GPU no matter how much you ask. A practical rule: assume you have about 80-90% of the nameplate figure available for weights plus KV cache. This is why an "8 GB" model card and an "8 GB" GPU are not a match — you want meaningful headroom, not a photo finish.

The bytes-per-parameter table, worked in full

The formula parameters × bits-per-parameter ÷ 8 is worth internalizing as a lookup table, because it lets you convert any model's parameter count and any quant level into a file size in your head. Here is the bytes-per-parameter figure at each common precision, and what a 7B and a 70B model weigh at each:

Precision Bits/param Bytes/param 7B model 70B model
FP32 (full) 32 4.0 ~28 GB ~280 GB
FP16 / BF16 16 2.0 ~14 GB ~140 GB
Q8_0 ~8.5 ~1.06 ~7.4 GB ~74 GB
Q6_K ~6.6 ~0.82 ~5.7 GB ~57 GB
Q5_K_M ~5.7 ~0.71 ~5.0 GB ~50 GB
Q4_K_M ~4.8 ~0.60 ~4.2 GB ~42 GB
Q3_K_M ~3.9 ~0.49 ~3.4 GB ~34 GB
Q2_K ~3.4 ~0.42 ~3.0 GB ~30 GB

Two things are worth noticing. First, the "bits per parameter" for a k-quant is not exactly the number in the label — a Q4_K_M averages closer to 4.8 bits per weight, not 4.0, because the method keeps some tensors (attention layers, embeddings) at higher precision where accuracy matters most and squeezes the rest harder. The label is a family name, not a literal bit count. Second, notice that Q2_K barely saves space over Q3_K_M while costing far more quality — a concrete illustration of why the bottom of the range is a bad deal. You give up a lot of intelligence to shave a few hundred megabytes.

To turn a file size into a fit decision, add the KV cache (covered next) and overhead. A Q4_K_M 7B model at ~4.2 GB, plus ~1 GB of KV cache at a moderate context, plus ~1 GB of runner overhead, lands around 6 GB of live memory — which is why 8 GB is the honest floor for the 7-9B tier, not the 4-5 GB the weights alone suggest.

Context length is not free: the KV cache

The single most overlooked line item in local LLM memory is the KV cache. When a model processes text, it computes and stores a "key" and "value" vector for every token at every layer, so it doesn't have to recompute the whole history for each new token. That cache lives in memory alongside the weights, and it grows linearly with context length. Feed the model twice as much text and the cache doubles.

The rough size is: 2 × layers × context_length × hidden_dim × bytes_per_element. The exact numbers depend on the architecture, but the shape is what matters — it scales with how long a conversation or document you hold in context. For a mid-size model, the KV cache can run from a few hundred megabytes at a short 4K context to several gigabytes at 32K or beyond. On a memory-tight setup, this is frequently what tips a model from "fits" to "out of memory" the moment you paste in a long document.

Two levers help. Modern architectures use grouped-query attention (GQA), which shares key/value heads across query heads and cuts the cache size several-fold compared with older full multi-head attention — one reason newer models handle long context more gracefully on the same hardware. And most runners let you quantize the KV cache itself (for example to 8-bit or 4-bit), roughly halving or quartering its footprint at a modest quality cost, which is often the cheapest way to buy back headroom for a longer context. The practical upshot: when someone says "this model supports 128K context," that is the model's trained ceiling, not a promise your hardware can hold 128K tokens in memory. You set the context window you can afford, and it is usually far below the maximum.

Quantization: the trick that makes it possible

Quantization is why any of this works on consumer hardware, so it's worth understanding rather than treating as a magic slider.

A freshly trained model stores each weight as a 16-bit (or 32-bit) floating-point number. That's precise but bulky. Quantization stores those same weights using fewer bits — 8, 6, 5, 4, 3, or even 2 — accepting some rounding error in exchange for a smaller, faster model. It's lossy compression for neural networks.

The crucial, non-obvious fact is that the quality loss is non-linear. Going from 16-bit down to about 4-bit costs surprisingly little — often a few percent on benchmarks, frequently unnoticeable in casual use. But below roughly 3 bits, quality falls off a cliff: the model starts making dumb errors, losing coherence, and forgetting instructions. This is why the community converged on 4-bit-ish quants as the default: it's the knee of the curve, the best trade of size for smarts.

You'll see cryptic labels like Q4_K_M, Q5_K_M, Q6_K, Q8_0. Don't memorize them; learn the pattern:

  • The number is the approximate bits per weight. Higher = bigger and better.
  • K denotes the modern "k-quant" method, which is smarter about spending bits where they matter.
  • The suffix (S/M/L for small/medium/large) is a fine-tuning of the size/quality trade within a level.

A durable rule of thumb: prefer a larger model at a lower quant over a smaller model at a higher quant, down to about 4-bit. A 13B model at Q4 will usually beat a 7B model at Q8, despite similar file sizes. Intelligence lives more in parameter count than in precision, until precision gets too low. If you want the full picture of open models and their trade-offs, the open-weights guide goes deeper on licensing and model families.

Why is the curve non-linear? Weights in a trained network are not equally important, and most of them are small. Quantization introduces a rounding error roughly proportional to the step size between representable values. At 8 or 6 bits there are enough distinct levels that the error is tiny relative to the signal. At 4 bits it's noticeable but the important weights — the ones the k-quant method deliberately keeps at higher precision — are protected, so the model stays coherent. Below 3 bits there simply aren't enough levels left to represent the weight distribution, the protected-tensor trick can't compensate, and errors compound across dozens of layers into visibly worse output. The "cliff" is the point where accumulated rounding error overwhelms the model's redundancy.

A related nuance: quantization hurts bigger models less. A 70B model has more redundancy to absorb rounding error, so it tolerates aggressive 3-bit quantization far better than a 3B model does — a 3B at Q3 can be nearly unusable while a 70B at Q3 remains respectable. This interacts with the size-versus-precision rule above: the larger the model, the more comfortable you can be trading precision for the parameter count you actually want. And be aware that perplexity, the number often quoted to measure quantization damage, understates the real-world hit on structured tasks — code generation and exact instruction-following degrade faster than a small perplexity delta suggests, so treat a "only 2% worse" claim with mild suspicion for anything precise.

Quant formats beyond GGUF: GPTQ, AWQ, bitsandbytes, MLX

GGUF is the dominant format for CPU-and-consumer-GPU inference, but it is not the only quantization scheme, and the differences matter once you move toward GPU-first serving. The landscape divides by when and how the quantization happens.

  • GGUF (llama.cpp k-quants) is a post-training quantization designed for mixed CPU/GPU inference. It stores different tensors at different bit widths and runs anywhere. It is the right default for individuals and the format almost every consumer runner speaks.
  • GPTQ is a post-training method that quantizes one layer at a time while minimizing the reconstruction error against a small calibration dataset. It targets GPU inference and typically produces 4-bit weights that run fast on CUDA. Because it uses calibration data, its quality can edge out naive rounding, but it is GPU-oriented and less flexible about partial CPU offload.
  • AWQ (Activation-aware Weight Quantization) also uses calibration, but its insight is to protect the weights that correspond to the largest activations — the channels that actually move the output — rather than treating all weights equally. In practice AWQ often preserves quality at 4-bit slightly better than GPTQ and is popular for serving with high-throughput engines.
  • bitsandbytes provides on-the-fly 8-bit and 4-bit (NF4) quantization inside the PyTorch/Transformers stack. It is less about deployment and more about loading a model that wouldn't otherwise fit for experimentation and, crucially, for QLoRA fine-tuning (more below). It is convenient rather than maximally fast.
  • MLX is Apple's array framework; MLX-format models are quantized specifically for Apple Silicon's unified memory and Metal GPU, and on a Mac an MLX build often runs faster than the equivalent GGUF because it is tuned to that hardware path.

The practical guidance: if you are running on a laptop or a single consumer GPU through Ollama or LM Studio, you will use GGUF and rarely think about the rest. If you graduate to serving a model to multiple users on a rented GPU, GPTQ or AWQ through a batching engine becomes relevant. If you are on a Mac and want the last 20% of speed, look for an MLX build. And if you are fine-tuning, bitsandbytes' NF4 is the format that makes it possible on modest hardware.

GGUF and the tooling tiers

Two things you download: a format and a runner.

The dominant format for local inference is GGUF — a single self-contained file holding the quantized weights, the tokenizer, and metadata. You download one .gguf file at your chosen quant and point a runner at it. (Other formats exist for GPU-first, unquantized serving, but GGUF is the lingua franca of consumer local LLMs.)

The runners fall into three tiers. Pick the tier that matches how much control you want; the specific brand is interchangeable and will change over time.

Tier 1 — one-line launchers. Tools in this category (Ollama is the current archetype) hide everything. You run one command, it downloads a sensible quant, and you're chatting. It also exposes a local API endpoint that mimics the common cloud API shape, so apps can talk to your local model as if it were a hosted one. Start here.

Tier 2 — GUI apps. Tools like LM Studio give you a desktop interface: browse and download models, see estimated memory fit before you commit, tweak parameters with sliders, and chat in a familiar window. Best if you dislike the terminal or want to experiment with settings visually.

Tier 3 — the engine. Underneath most of the above sits llama.cpp, the C/C++ inference engine that made efficient CPU-and-GPU local inference practical. You can use it directly for maximum control, custom flags, and scripting. For GPU-heavy or multi-user serving there are separate high-throughput engines optimized for batching — that's a different, more advanced lane. Most individuals never need to leave Tier 1 or 2.

A concept worth knowing across all tiers is offload: splitting the model between GPU and CPU. If a model is slightly too big for your VRAM, you can offload most layers to the GPU and keep the rest on the CPU. It runs — just slower, bottlenecked by the CPU-resident layers. This is the escape hatch that lets you run a model a size above what your VRAM alone would allow. The penalty is not gentle, though: because generation reads every layer for every token, even a handful of CPU-resident layers can halve your token rate, since the GPU now waits on the slow memory path each pass. Offloading 90% of layers does not get you 90% of full-GPU speed — expect closer to the speed of whatever fraction is stuck on the CPU. Use it to make something run, not to make something fast.

The runtime landscape, compared

The three-tier mental model is enough to get started, but if you are choosing deliberately it helps to know what each major runtime is actually optimized for. They are not interchangeable at the margins — they make different bets about hardware and use case.

  • llama.cpp is the foundational engine: a portable C/C++ implementation that runs GGUF models across CPU, CUDA, Metal, Vulkan, and more. Its strength is reach — it runs on almost anything, including a Raspberry Pi — and efficient CPU-plus-partial-offload inference. Most consumer tools are wrappers around it. It is single-user-focused; it does not do heavy request batching.
  • Ollama wraps llama.cpp with a model registry, automatic quant selection, a background server, and a clean API. It optimizes for getting one person chatting in one command. It is the right first tool for almost everyone and increasingly capable of serving small teams.
  • LM Studio is a polished desktop GUI, also llama.cpp-based (with an MLX path on Macs), aimed at people who want to browse models, see memory-fit estimates, and tune parameters visually without a terminal. It also exposes a local API server.
  • vLLM is a different animal: a GPU-first, high-throughput serving engine built around PagedAttention, which manages the KV cache like virtual memory so it can batch many concurrent requests efficiently. For a single user it is not faster than llama.cpp and it demands a proper GPU with enough VRAM to hold the model unquantized or lightly quantized (GPTQ/AWQ). Its payoff is throughput — dozens of simultaneous requests — which matters for a shared internal service, not a laptop.
  • MLX (and mlx-lm) is Apple's framework for Apple Silicon. On a Mac it extracts the most from the unified-memory architecture and Metal, often beating GGUF on tokens per second for the same model. If your whole world is a Mac, it is worth trying alongside Ollama.
  • TensorRT-LLM sits at the far end: NVIDIA's heavily optimized engine that compiles models for specific GPU architectures to squeeze out maximum performance. It offers the best speed on supported NVIDIA hardware at the cost of setup complexity and portability. This is data-center and serious-workstation territory.

The decision rule stays simple. One person, any hardware, minimal fuss: Ollama or LM Studio. A Mac chasing speed: MLX. Serving many users off a GPU: vLLM, or TensorRT-LLM if you need to wring out every last token per second and can pay the setup tax. Everything else is a variation on these.

Serving locally: API compatibility and tool use

The feature that turns a local model from a novelty into infrastructure is the local API server. Ollama, LM Studio, llama.cpp's own server, and vLLM all expose an HTTP endpoint that mimics the widely-adopted OpenAI-compatible chat API shape. This is a big deal in practice: it means any app, plugin, or script written to talk to a hosted chat API can be pointed at localhost instead by changing a base URL — no code rewrite. Editor plugins, note-taking tools, browser extensions, and agent frameworks mostly speak this dialect, so your local model drops into the existing ecosystem.

Two capabilities are worth understanding before you rely on them locally:

  • Structured output / JSON mode. Many runners can constrain generation to valid JSON or to a supplied schema, using grammar-based sampling (llama.cpp calls this GBNF grammars) that only allows tokens consistent with the format. This is more reliable locally than "please respond in JSON" prompting, because the constraint is enforced at the sampler, not requested politely. If you are building anything that parses model output, use it.
  • Tool / function calling. The OpenAI-compatible API includes a tool-calling convention, and many runners now support it — the model emits a structured request to call a named function, your code runs it, and you feed the result back. The catch is that tool-calling reliability is a function of the model, not just the runner: smaller local models call tools less dependably, hallucinate arguments more often, and lose the thread over multi-step chains sooner than frontier models. It works, but budget for more validation and retries than you would with a hosted model. This is one of the areas where the local-versus-cloud capability gap is most visible.

For anything beyond a single machine, remember the server is a service you are now responsible for: bind it to localhost unless you deliberately want network access, and if you do expose it, put authentication in front of it. An unauthenticated inference endpoint on a shared network is an open door.

Fine-tuning and LoRA on your own machine

You can not only run models locally, you can adapt them locally — and this is more accessible than most people assume, thanks to two techniques. The context here is that full fine-tuning (updating every weight) requires memory for the weights, their gradients, and the optimizer state — often more than 10x the model's inference footprint, which is why it needs data-center GPUs.

LoRA (Low-Rank Adaptation) sidesteps this. Instead of updating the billions of original weights, it freezes them and trains a small pair of low-rank matrices that inject a learned adjustment into each layer. You end up training a tiny fraction of the parameters — often well under 1% — which slashes the memory and compute needed. The result is a small "adapter" file (megabytes, not gigabytes) that you load on top of the base model at inference time. You can keep several adapters for different tasks and swap them without duplicating the base model.

QLoRA goes further by loading the frozen base model in 4-bit (via bitsandbytes NF4) while training the LoRA adapter in higher precision on top. This combination is what put fine-tuning within reach of a single consumer GPU: you can adapt a 7B model on a card with 12-16 GB of VRAM, and larger models on a 24 GB card. Tooling like Hugging Face's PEFT library, and higher-level wrappers such as Axolotl and Unsloth, package this into a config-file workflow. Our walkthrough on how to fine-tune a model covers the process end to end.

Be realistic about what fine-tuning does, though. It is excellent for teaching a model a style, a format, or a narrow domain's vocabulary and conventions — making outputs consistently match your house voice, or reliably produce a specific structure. It is a poor and expensive way to teach a model new facts; for that, retrieval (feeding relevant documents into the context at query time) is usually cheaper, more current, and more honest, since the model cites what it was given rather than half-remembering it. Reach for a local fine-tune when you need behavior, not knowledge.

Hardware: what you actually need

You don't need a server. You need to be honest about which tier you're targeting.

  • Any modern laptop (integrated graphics, 16 GB RAM): Comfortable with 7-9B models at 4-bit. Genuinely useful for drafting, summarizing, coding help, and Q&A over your own notes.
  • Apple Silicon Mac (16-64 GB unified memory): The best value in local LLMs. Unified memory means the GPU can use most of your RAM, so a 32 GB Mac runs models that would need an expensive dedicated GPU on a PC.
  • PC with a dedicated GPU: VRAM is the number that matters, not the marketing name. 8 GB handles 7-9B comfortably; 12-16 GB reaches into the teens-of-billions; 24 GB opens up ~30B-class models. Two GPUs or a workstation card get you to 70B.
  • CPU-only, no GPU: It works. It's slow — think a handful of tokens per second on bigger models — but for occasional, non-interactive tasks (batch summarizing overnight), it's fine and free.

Speed is measured in tokens per second. Above ~15-20 tok/s feels conversational; below ~5 tok/s feels like waiting. Don't obsess over benchmarks — a model that's "smart but 2 tok/s" is often less useful in practice than one that's "decent but instant," because you'll actually use the fast one.

Two hardware details are worth flagging because they trip people up. First, on the PC-GPU path, memory bandwidth matters as much as VRAM capacity — a card with 16 GB of slow memory can generate tokens slower than one with 12 GB of fast memory, because generation is bandwidth-bound. Capacity decides what fits; bandwidth decides how fast it runs. Read both numbers on a spec sheet, not just the gigabytes. Second, on the Apple Silicon path, the unified-memory advantage is real but the memory-bandwidth tier varies enormously across the chip lineup — the base chip has a fraction of the bandwidth of the top "Max" and "Ultra" variants. Two Macs with the same total RAM can differ several-fold in token generation speed depending on which tier of chip they carry. When people say "Macs are great for local LLMs," they mostly mean the higher-bandwidth chips; the base models are fine for small models but not the giant-killers the reputation implies.

A note on the awkward middle: mixing consumer GPUs to reach more VRAM works but adds complexity and communication overhead, and two 12 GB cards do not equal one 24 GB card in convenience even if the capacity matches. If you are seriously targeting 70B-class models, a single high-VRAM card or a high-bandwidth, high-memory Mac is usually less painful than a multi-GPU rig, unless you already have the hardware.

Troubleshooting speed: why it's slow and what to do

When a local model runs slower than you expected, the cause is almost always one of a short list. Diagnose in this order:

  • The model is spilling out of VRAM into system RAM. This is the number-one culprit and the biggest cliff. If even a few layers or the KV cache don't fit on the GPU, they fall back to slow CPU memory and drag the whole thing down. Symptoms: dramatically slower than expected, and it gets worse as the conversation grows (because the KV cache is growing into the overflow). Fix: use a smaller quant, a smaller model, or a shorter context so everything fits in VRAM with headroom. Confirm by watching whether all layers report as GPU-offloaded.
  • You're actually running on CPU. More common than you'd think — the GPU build wasn't installed, the runner didn't detect the GPU, or drivers are stale. Symptoms: uniformly slow regardless of model size, GPU utilization near zero. Fix: verify the runner sees your GPU and that you installed the GPU-enabled build (CUDA, Metal, ROCm, or Vulkan as appropriate).
  • The context is huge. A long conversation or a big pasted document means a large KV cache and a long prefill. Symptoms: the first token takes a long time (prefill), or memory pressure appears only after you paste something big. Fix: trim the context, start a fresh session, or quantize the KV cache.
  • You're memory-bandwidth-limited and that's just the ceiling. If the model fits, the GPU is engaged, and the context is reasonable, then the speed you're getting may simply be the hardware's honest limit (recall: token rate ≈ bandwidth ÷ model bytes). Fix: a smaller or more aggressively quantized model is the only lever left short of new hardware.
  • Thermal throttling on a laptop. Sustained generation heats the chip; fanless or thin laptops throttle after a minute and slow down. Symptoms: fast at first, then degrades and stays degraded. Fix: better cooling, or accept it for short interactions.

The meta-lesson: before blaming the model or the tool, confirm the model fully fits in fast memory with the GPU actually doing the work. That single check resolves the large majority of "why is local so slow" complaints.

Privacy and security: the real benefits and the real limits

Privacy is the headline reason to run locally, and it is a genuine, categorical difference — not a marketing gradient. With true local inference, your prompt is transformed into tokens, multiplied through the weights, and turned back into text entirely within your machine's memory. There is no network request, so there is nothing to intercept, log, retain, or subpoena. For regulated data (health, legal, financial), air-gapped environments, and anyone who simply doesn't want their thinking on someone else's server, this is the whole ballgame, and it is worth stating plainly because cloud providers' privacy promises, however sincere, are policies you must trust rather than physics you can verify. The broader trade-offs are covered in our note on AI and privacy.

But be precise about the boundaries, because "local" gets oversold:

  • The tool is only private if inference is actually local. Some "local" apps quietly fall back to a cloud API for larger models, sync settings or history to a server, or phone home with telemetry. Verify. The honest test is whether the thing keeps working with your network cable unplugged — if it does, inference is genuinely on-device.
  • Local privacy is not local security. The model file you downloaded is data, but the runner is software with network capability and file access. Download models and tools from reputable sources; a malicious build or a tampered model file is a real supply-chain risk. And if you expose the local API server to your network without authentication, you have created an open endpoint that anyone on that network can use.
  • Privacy is not safety or accuracy. A local model with no cloud safety layer will more readily produce wrong, biased, or harmful output, and it hallucinates just as confidently as any other model. Running it privately means you are the only reviewer. Off-device privacy does not make on-device output trustworthy — you still have to check it.
  • Your data can still leak through what you build. If you wire a local model into a tool that then sends its outputs somewhere — an app that emails a summary, a plugin that posts to a server — the privacy boundary is only as tight as the weakest link in that chain, not the model itself.

The accurate framing: local inference gives you data locality — a strong, verifiable guarantee that your inputs stay on your machine during inference. It does not automatically give you a secure system or a trustworthy answer. Those are separate responsibilities you still own.

When local genuinely beats the cloud (and when it doesn't)

Skepticism cuts both ways: local isn't a moral victory, and cloud isn't a scam. They're different tools.

Local wins when:

  • Privacy is non-negotiable. Sensitive documents, regulated data, personal journaling, confidential code. Nothing leaves the machine, full stop. This is the strongest case, and it's covered more broadly in our note on AI and privacy.
  • You're offline or want to be. Planes, remote fieldwork, air-gapped environments, or just not wanting a dependency on someone else's uptime.
  • Volume is high and repetitive. Classifying thousands of items, bulk summarizing, or a background feature that runs constantly — per-token cloud billing adds up, while local is a fixed hardware cost. The economics of inference cost explains why this crossover exists.
  • You're learning or building. Tinkering with prompts, fine-tunes, or agent loops without a meter running is liberating.

Cloud wins when:

  • You need peak intelligence. The hardest reasoning, the longest contexts, the most reliable instruction-following still live in frontier models you can't fit on a laptop.
  • You want zero setup and always-current models. No downloads, no memory math, always the latest version.
  • Your workload is spiky. Occasional heavy use is cheaper to rent than to buy hardware for.

The honest answer for most people is both: a local model for the private, high-volume, or offline 80%, and a cloud model for the hard 20%. Deciding which model for which job is its own skill — our guide on which AI to use helps with that split.

There's also a subtle economic point people skip. "Local is free" ignores the capital cost of the hardware and the electricity, and it ignores your time. A dedicated 24 GB GPU is a real expense that would buy a lot of API calls at frontier-model prices. Local's cost advantage is genuine but conditional: it wins decisively when volume is high and sustained (the fixed hardware cost amortizes over millions of tokens), when you already own the hardware (marginal cost is just electricity), or when the value is privacy rather than dollars. For occasional, spiky use, renting compute is almost always cheaper than owning it — the same logic as any capacity-planning decision. The economics of inference cost works through where that crossover actually sits.

Common misconceptions

A handful of beliefs cause most of the disappointment and confusion around local LLMs. Clearing them up saves time and manages expectations:

  • "A local model is basically the same as the cloud chatbot, just on my computer." No. You are running the raw model, not the product. The hosted assistant adds retrieval, memory, tool integration, safety systems, and usually a far larger model. A local 8B model is capable but noticeably less smart than a frontier model, and it comes with none of the surrounding scaffolding unless you build it.
  • "More parameters always means better." Only up to the point where it still fits and runs at a usable speed. A 70B model that generates at 2 tok/s because it's overflowing your VRAM is worse in practice than a snappy 8B you'll actually use. The best model is the largest one that runs fast enough for your workflow, not the largest one that technically loads.
  • "Higher quant is always worth it." Above ~5-6 bit the quality gains are marginal while the memory and speed costs are real. For most people Q4_K_M or Q5_K_M is the sweet spot; going to Q8 mostly buys you a bigger file and slower generation for a difference you won't notice.
  • "128K context means I can use 128K context." That's the model's trained maximum, not what your hardware can hold. The KV cache for a full long context can exceed the weights. You run the context window you can afford in memory, which is usually a fraction of the ceiling.
  • "Local means secure." It means private during inference, which is a data-locality guarantee, not a security posture. The runner is still software, the model file is still a download from somewhere, and an exposed API server is still an exposed server.
  • "CPU-only is pointless." For interactive chat it's slow, but for non-interactive, batch work — summarizing a folder of documents overnight, classifying a backlog — a few tokens per second running for free while you sleep is perfectly useful.
  • "Fine-tuning will teach it my company's facts." Fine-tuning teaches behavior and style well and facts poorly. For knowledge, retrieval beats fine-tuning on cost, freshness, and honesty.

A sane way to get started

Skip the analysis paralysis. A reliable first path:

  1. Install a Tier-1 launcher. One command, no configuration.
  2. Pull a 7-9B instruct model at a 4-bit quant. It'll fit almost anywhere and gives you an honest feel for local quality.
  3. Chat with it for real work — summarize a document, draft an email, ask it to explain code. Notice where it's great (fast, private, tireless) and where it stumbles (subtle reasoning, obscure facts).
  4. Only then size up or down. If it's too slow, drop a tier. If it's not smart enough and you have the memory, jump to a 13-14B model. Re-run the memory math each time.
  5. Point a tool at the local API. Once you're comfortable, wire the local endpoint into a note-taking app, an editor plugin, or a small script. That's where local stops being a toy and becomes infrastructure.

Getting better output is the same craft as with any model; the fundamentals in how to write better prompts apply unchanged whether the model is on your laptop or a server farm. If anything, they matter more locally, because a smaller model has less slack to recover from a vague prompt.

FAQ

Is a local LLM as good as a frontier cloud model? No, and it's worth being blunt about it. A model that fits on consumer hardware is smaller and less capable than the best hosted models at hard reasoning, long-context work, and obscure knowledge. What it offers instead is privacy, offline access, no per-token cost, and full control. For a large share of everyday tasks — drafting, summarizing, formatting, simple Q&A, coding help — that gap is small enough not to matter. For the hardest problems, it's real. Use the right tool for each.

What's the minimum hardware to run an LLM locally? Less than you'd think. A laptop with 16 GB of RAM can comfortably run a 7-9 billion parameter model at 4-bit quantization, and even 8 GB machines can run smaller 1-3B models. A dedicated GPU or an Apple Silicon Mac makes it faster and lets you run bigger models, but neither is required to start. CPU-only inference works; it's just slower.

What is quantization and which level should I pick? Quantization stores a model's weights using fewer bits to shrink its size and memory needs, at the cost of some precision. The quality loss is minor from 16-bit down to about 4-bit, then degrades sharply below 3-bit. For most people a 4-bit quant (often labeled Q4_K_M) is the best default — the knee of the size-versus-quality curve. Go higher (5-6 bit) if you have memory to spare and want maximum fidelity; avoid going below 3-bit.

What's the difference between Ollama, LM Studio, and llama.cpp? They're three tiers of the same idea. llama.cpp is the underlying inference engine that does the actual work. Ollama is a one-line launcher built for simplicity — install, run one command, start chatting, with a built-in local API. LM Studio is a graphical desktop app for browsing, downloading, and chatting with visual controls. Beginners should start with a launcher or the GUI; the engine is there when you want full control.

Do my prompts stay private with a local model? Yes. When you run a model locally, inference happens entirely on your own hardware and nothing is sent over the network. Your prompts, documents, and outputs never reach an external server. That's the single biggest reason to run locally, and it's genuinely different from cloud assistants where your inputs are transmitted and may be logged or used per the provider's policy.

Can I run a local model on my phone? Increasingly, yes, for small models. Phones with enough RAM can run 1-3 billion parameter models at low quants, which are fine for autocomplete, simple rewriting, and basic Q&A. Don't expect laptop-class quality — the memory and thermal limits are real — but fully offline, on-device AI on a phone is no longer science fiction.

Why is my local model so slow even though it "fits"? The most common reason is that it doesn't actually fit in fast memory — a few layers or the growing KV cache have spilled into slow system RAM, which drags everything down. The second most common is that you're inadvertently running on the CPU because the GPU-enabled build isn't installed or the runner didn't detect your GPU. Check that every layer is offloaded to the GPU and that GPU utilization is non-zero during generation. If both are fine and it's still slow, you may simply be hitting your hardware's memory-bandwidth ceiling, in which case a smaller or more aggressively quantized model is the only fix short of new hardware. A long context also slows the first token (prefill) and grows the cache, so trimming it helps.

Can I fine-tune a model locally, and when should I? Yes, using LoRA and especially QLoRA, which loads the base model in 4-bit and trains a small adapter on top — this fits a 7B fine-tune on a 12-16 GB GPU. Reach for it when you need the model to consistently match a style, format, or narrow domain's conventions. Don't reach for it to teach the model new facts; retrieval (feeding relevant documents into the prompt at query time) is cheaper, stays current, and is more honest because the model works from what you gave it rather than half-remembering training data. Fine-tuning is for behavior, retrieval is for knowledge.

How does context length affect what I can run? It affects memory directly through the KV cache, which grows linearly with the number of tokens in context. A model that fits comfortably at a 4K context can run out of memory at 32K, because the cache can grow to rival the weights themselves. If you need long context on tight memory, pick a model with grouped-query attention (most modern ones), quantize the KV cache to 8-bit or 4-bit in your runner's settings, and set the context window to what you can actually afford rather than the model's advertised maximum.

The durable version

The tools will be renamed, the model leaderboard will reshuffle, and next year's laptop will run what needs a workstation today. None of that touches the core skill. If you can size a model against your memory, choose a quant on the right side of the quality cliff, and pick the tooling tier that matches how much control you want, you can adopt any new local model as it appears — and decide, task by task, whether the private machine on your desk or the smart one in the cloud is the right one to ask.