Prompt20
NewsDataBlog
All posts
context-engineeringpromptingragagentsmemorytoken-budgethow-toevergreen

Context Engineering: Managing What the Model Actually Sees

The discipline past prompt-writing: assembling, compressing, and ordering everything in the context window — retrieval, tools, memory, history — within a token budget. Why it's the real skill behind good agents.

By Prompt20 Editorial · 28 min read

A model has no memory, no filing cabinet, and no idea who you are. Everything it knows about your specific problem — the question, the relevant documents, the earlier conversation, the tools it can call, the rules it must follow — has to be present, in text, inside a single fixed-size buffer called the context window at the moment it generates a response. Context engineering is the discipline of deciding what goes into that buffer, in what order, and how much space each thing gets. Prompt writing asks "how do I phrase this request?" Context engineering asks "what should the model be looking at when it reads this request at all?"

That shift — from phrasing to assembly — is why most serious AI work is now a systems problem rather than a copywriting one. A single well-worded prompt still matters, but in any real application the prompt is a small part of what fills the window. The rest is retrieved passages, tool outputs, conversation history, structured instructions, and memory carried between sessions. Someone, or some pipeline, has to choose all of it. Do that badly and even a frontier model looks stupid: it answers the wrong question, ignores the document you pasted, or forgets an instruction from four turns ago. Do it well and a mid-tier model can feel sharp. This guide is about doing it well.

Key takeaways

  • Context is everything the model sees at inference time, not just your prompt: system instructions, retrieved data, tool results, history, and memory all share one window.
  • The window is a budget, not a backpack. Every token you add competes for space and attention with every other token. More context is not automatically better.
  • Position matters. Models attend unevenly across a long input — material at the very start and very end tends to land harder than material buried in the middle.
  • Relevance beats volume. A few precisely-chosen passages usually outperform dumping everything and hoping the model sorts it out.
  • Agents live or die on context. An agent is a loop that keeps rewriting its own context; managing that growth — compressing, pruning, summarizing — is the core engineering challenge.
  • It is measurable. Unlike prompt "vibes," context decisions can be tested: swap a retrieval strategy, hold everything else fixed, and watch the eval scores move.

Table of contents

Why "prompt engineering" quietly became context engineering

For a couple of years, "prompt engineering" was the headline skill of the generative-AI era. The framing made sense when the interaction was a person typing into a box and reading what came back: the only variable you controlled was the wording, so wording was where all the craft went. Communities traded "magic phrases," role-play preambles ("you are a world-class expert…"), and formatting tricks, and some of them genuinely helped. But the framing quietly stopped describing what practitioners were actually doing.

The shift happened because the interesting systems stopped being single messages. As soon as you attach a knowledge base, a set of tools, a running conversation, and memory that persists across sessions, the wording of any one message becomes a rounding error next to the question of what else is in the window when the model reads it. You can spend an afternoon polishing a sentence, or you can fix the fact that your retrieval step is returning the wrong three paragraphs — and the second one moves the needle a hundred times more. The center of gravity moved from the phrasing of the request to the composition of the input.

That is the whole thesis of context engineering, and it is worth stating bluntly because it inverts an intuition many people still carry: the model is usually not the bottleneck, and the prompt is usually not the bottleneck. The context is. Most production failures — the confident wrong answer, the ignored document, the forgotten instruction — are not the model being dumb or the prompt being clumsy. They are the model faithfully answering from a window that was assembled badly. Prompt engineering optimizes one component of that window. Context engineering owns the whole assembly, which is why it subsumes prompt engineering rather than replacing it: good wording is still one of the levers, just no longer the only one, or even the biggest one.

None of this makes prompting obsolete. The instructions you write are still part of the context, and writing them clearly still matters — the habits in how to write better prompts apply verbatim to the instruction layer. What changed is the scope of the job. You are no longer writing a message; you are curating an information environment, one that is rebuilt from scratch on every single call.

What "context" actually contains

When people picture a chatbot, they imagine typing a message and getting a reply. Under the hood, the model never sees just your message. It sees a single concatenated blob of text assembled fresh for every generation. That blob typically includes several distinct layers:

  • System instructions — the standing rules, persona, and constraints set by whoever built the application. You usually can't see these, but they're first in line.
  • Tool and function definitions — descriptions of what the model is allowed to call (search, code execution, a database query) and how to format those calls.
  • Retrieved knowledge — passages pulled from documents, a knowledge base, or the web, injected to give the model facts it wasn't trained on or can't be trusted to recall.
  • Conversation history — the running transcript of the current session, so the model appears to "remember" what you said earlier.
  • Long-term memory — facts carried across sessions (your name, preferences, prior decisions), stored outside the model and re-injected when relevant.
  • The current user message — the actual thing you just asked.

All of this is text, all of it counts against the same token budget, and all of it competes for the model's attention. If you understand how a chatbot turns this blob into a reply, the rest follows naturally — we cover that machinery in how AI chatbots work. Context engineering is the practice of curating these layers deliberately instead of letting them pile up by accident.

The anatomy of the context window

It's worth slowing down on the structure, because the layers are not interchangeable — each has a different source, a different owner, a different failure mode, and a different position in the assembled input. The context window is the fixed-size buffer the whole thing lives in; if you want the mechanics of why it's fixed and how its size is measured in tokens, the context window explainer and the tokenization primer cover the ground this section assumes. Here we care about what fills it.

A production context, read top to bottom, is usually assembled from these slabs:

  • The system prompt. The outermost frame: identity, tone, hard constraints, safety rules, output-format contracts. It is set by the application, not the user, and it typically sits first. Because early positions are weighted heavily, this is prime real estate — but it's also static, which means every wasted sentence here is a tax paid on every single call forever. Ruthless economy in the system prompt pays compounding dividends.
  • Task instructions. The specific directions for this class of request — the rubric, the steps, the definition of done. Sometimes fused with the system prompt, sometimes injected per-task. The distinction matters when one application serves many task types from one base persona.
  • Few-shot examples. Worked demonstrations of the input-output mapping you want. Examples are often the highest-leverage tokens in the window: a model that ignores a paragraph of instructions will frequently obey two clean examples, because a demonstration removes ambiguity that prose can't. The cost is real, though — every example is tokens you pay for on every call, so the engineering question is how few examples buy the behavior, and whether they can be retrieved dynamically rather than hard-coded.
  • Retrieved documents. The facts fetched at query time from a corpus the model was never trained on, or can't be trusted to recall precisely. This slab is the most dynamic and the most dangerous: it changes every call, it's often the largest, and — crucially — it's frequently sourced from content you don't fully control.
  • Tool and schema definitions. The menu of actions the model may take and the exact JSON shapes it must emit to take them. These are instructions in disguise, and they compete for attention like everything else; a bloated tool schema quietly degrades reasoning about the actual task.
  • Long-term memory. Durable facts about the user or the project — preferences, prior decisions, standing context — stored outside the model and re-injected selectively. Memory is what makes an assistant feel continuous across sessions, and it's also a quiet source of staleness when a fact that was true in March is still being injected in July.
  • Conversation history. The running transcript of the current session. It grows monotonically unless something trims it, and it carries strong recency weight, which is a blessing (the model tracks the thread) and a curse (old-but-important instructions get buried under recent chatter).
  • The scratchpad. In reasoning and agentic setups, the model's own intermediate work — plans, notes, chain-of-thought, partial results — is appended back into the window so later steps can build on it. The scratchpad is context the model writes for itself, and managing its growth is a large part of agentic context engineering.
  • The current user message. The actual request, usually last, in a high-attention position by default.

Two properties of this stack drive almost every design decision. First, it is heterogeneous in trust: the system prompt is authored by you and trustworthy; a retrieved web page is authored by a stranger and is not. The model, however, sees one undifferentiated stream of tokens and extends the same credulity to all of it — the root of the security problem we return to later. Second, it is heterogeneous in volatility: the system prompt is frozen, history grows every turn, retrieval changes every query, memory drifts over weeks. A context strategy is really a set of policies for how each slab is sized, ordered, refreshed, and trimmed given those two axes.

The window is a budget, not a backpack

The most common mistake is treating the context window as free storage: if the model can hold 200,000 tokens, why not use them? Because the window behaves less like a backpack you keep stuffing and more like a budget you keep spending.

Three costs make this real. First, money and latency: you generally pay per token, and processing a large input takes longer, so a bloated context is slower and more expensive on every single call. The economics compound fast at scale — see AI inference cost economics for how those numbers actually add up.

Second, attention dilution: a model reading 100,000 tokens has to spread its attention across all of them. The signal you care about — the one clause that answers the question — is now surrounded by far more noise. Bigger inputs raise the odds the model latches onto something irrelevant.

Third, positional bias: models do not read a long context evenly. Empirically, content near the beginning and the end of the window tends to influence the output more than content in the middle. This is often called the "lost in the middle" effect. The practical upshot: where you place something matters almost as much as whether you include it. Put the critical instruction or the most relevant passage where the model is most likely to weight it, not buried in the middle of a giant paste.

So "just use the whole window" is rarely the right answer. The goal is the smallest context that fully supports a correct answer — dense, ordered, and relevant.

The practical discipline is to treat the token limit the way an accountant treats a fixed monthly income: allocate it deliberately across the slabs rather than letting whichever component happens to be verbose crowd out the rest. A useful mental model is a budget with reservations. Decide up front, for a given task, roughly how many tokens the system prompt and instructions get (fixed overhead), how many the retrieved passages get (the swing variable), how much recent history you'll always keep verbatim, and how much headroom you reserve for the model's output — a point that's easy to forget, because generation shares the same window and a context packed to the last token leaves no room to answer. When demand exceeds the budget, you don't silently overflow; you trigger a policy: retrieve fewer chunks, compress older history, or drop the lowest-priority slab. The failure mode to avoid is the one where a single unusually long tool result or a runaway conversation quietly consumes the space your instructions needed, and the model's behavior degrades for reasons that never show up in the prompt you wrote.

Budgeting also forces a healthy question about marginal value: for each slab, does the last thousand tokens you spent there measurably improve the answer? Often the fifth retrieved passage, the twelfth few-shot example, or the twentieth turn of history is pure cost — tokens that dilute attention and inflate the bill without changing the output. Context engineering is, in large part, finding and cutting those low-value tokens.

Prompt writing vs. context engineering

These are complementary, not competing, skills. Prompt writing is about the wording of a request; context engineering is about the environment that request lands in. If you've internalized the habits in how to write better prompts, context engineering is the next altitude up: you stop optimizing a sentence and start optimizing a pipeline.

Prompt writing Context engineering
Core question How do I phrase this? What should be in the window, and where?
Scope One request The whole assembled input, every turn
Main levers Wording, examples, tone, format Retrieval, compression, ordering, memory, pruning
Typical owner The end user The system / the developer
Fails by Vague or ambiguous instructions Missing, stale, buried, or bloated context
How you improve it Rewrite and re-read Measure, swap components, re-run evals

A useful way to hold the distinction: a great prompt inside a badly-assembled context still fails, because the model is looking at the wrong material. A mediocre prompt inside a well-assembled context often succeeds anyway, because everything it needs is right there and easy to attend to. That asymmetry is why context engineering is where the leverage is.

Retrieval: getting the right facts into the window

Most useful context isn't sitting in the conversation — it lives in documents, databases, tickets, or code the model has never seen. Retrieval is how you fetch the relevant slice and inject it. The dominant pattern is retrieval-augmented generation (RAG): given a query, search a corpus for the most relevant chunks, paste them into the context, and let the model answer grounded in that text rather than its parametric memory.

The quality of retrieval sets a ceiling on everything downstream. If your search returns the wrong three paragraphs, no amount of clever prompting recovers — the model is answering from bad source material and doesn't know it. This is why retrieval usually runs on embeddings and vector search: representing text as vectors so "find semantically similar passages" becomes a fast lookup. The mechanics are worth understanding directly; we cover them in the vector search and embeddings guide, and the production-grade version of the whole pipeline in RAG in production.

The context-engineering point is narrower than "build a RAG system." It's this: retrieval is a filter, and its job is to raise the relevance density of the window. Retrieve too little and the model lacks facts. Retrieve too much and you're back to attention dilution and lost-in-the-middle. Good retrieval returns few, high-quality, well-ordered chunks — and labels them clearly so the model knows what it's looking at.

Two mechanics inside retrieval are where relevance density is won or lost, and both are context decisions disguised as infrastructure decisions:

  • Chunking. Before anything can be retrieved, the corpus has to be cut into passages, and the size and boundaries of those passages set a hard ceiling on quality. Chunk too small and each passage loses the surrounding context that made it meaningful — a sentence about "the second option" retrieved without the list it referred to is worse than useless. Chunk too large and every hit drags in paragraphs of irrelevant filler, spending your budget on noise and burying the one relevant line in the middle of a block where the model underweights it. The boundaries matter as much as the size: splitting on semantic units (sections, functions, logical breaks) beats splitting on a blind character count, because a chunk that straddles two topics is a chunk that's half-wrong no matter which query pulls it.
  • Reranking. First-pass vector search optimizes for speed over precision; it casts a wide net and returns, say, the top 50 candidates by embedding similarity. Similarity is a coarse proxy for relevance, so those 50 are frequently in the wrong order, with genuinely on-point passages sitting at rank 30. A reranker — a slower, more accurate model that scores each candidate against the query directly — reorders them so the few you actually inject are the few that matter most. This two-stage pattern (cheap wide recall, then expensive precise ranking) is what lets you put three passages in the window instead of ten without losing the answer. It is one of the highest-leverage moves in the whole pipeline precisely because it improves relevance density without spending a single extra context token.

Both of these are covered end to end in the vector search and embeddings guide and the production RAG architecture; the reason they belong in a context-engineering discussion at all is that they determine what the model reads. A perfect prompt over badly-chunked, unranked retrieval is a perfect question asked of the wrong evidence.

Compression: fitting more signal in fewer tokens

When the material you need genuinely exceeds a sensible budget, the answer is compression, not a bigger paste. Several techniques trade a little fidelity for a lot of space:

  • Summarization. Replace a long transcript or document with a shorter recap that preserves the load-bearing facts. Lossy by design — the skill is knowing what's load-bearing.
  • Chunking and selection. Break sources into passages and include only the passages that scored well on retrieval, rather than whole documents.
  • Structured extraction. Convert prose into a compact form — a table, a JSON object, a bullet list of facts. Structure is denser than narration and often easier for the model to parse.
  • Deduplication. Strip repeated boilerplate, near-identical retrieved chunks, and redundant restatements. Repetition wastes budget without adding signal.

Compression is where judgment shows. Over-compress and you drop the one detail the answer hinged on; under-compress and you blow the budget. The reliable move is to compress the stale and general aggressively while keeping the fresh and specific verbatim — the exact figure, the precise error message, the literal clause someone asked about.

Ordering and formatting: same tokens, arranged better

Two windows can contain the identical set of facts and produce different answers, because the model doesn't read a bag of tokens — it reads a sequence, with structure. Ordering and formatting are the cheapest levers in context engineering: they change nothing about what you include and often move quality more than adding content would.

Start with order, because of the positional bias described above. Given that the beginning and end carry the most weight, the arrangement of the slabs is a decision, not a default. A common, defensible layout puts stable high-priority material (system instructions, the task rubric) at the very top, the volatile retrieved evidence in the middle, and the immediate question plus any final "answer using only the above" instruction at the very bottom — so the two things you most need the model to honor, the rules and the question, sit at the two high-attention ends, and the bulky evidence occupies the low-attention middle where its sheer volume does the least damage. Within the retrieved block, order matters too: if a reranker has told you which passage is most relevant, don't bury it at position four; put the strongest evidence where attention is highest. There's a subtle interaction with generation here — instructions placed after the evidence are the last thing the model reads before it starts writing, which is often exactly where you want the binding constraint.

Formatting is the second lever, and it works because structure carries meaning the model can exploit. Clearly delimiting the slabs — headers, XML-style tags, labeled sections — helps the model tell "these are your rules" from "this is a document I retrieved" from "this is what the user said," which both improves accuracy and, not incidentally, is a mild defense against injected text masquerading as instructions. Dense formats beat prose for reference data: a table of five records is easier for the model to read and cheaper in tokens than the same five records written as paragraphs. Consistent labeling of sources ("Source: internal wiki, updated 2026-02") lets the model reason about provenance and recency instead of treating every claim as equally authoritative. None of this is cosmetic. The same facts, delimited and ordered well, are simply easier for the model to use correctly — and easier is the entire game.

Tool results as structured context

The moment a model can call tools, a new and unusually messy source starts feeding the window: the results those tools return. A web search dumps a page of snippets; a database query returns rows; an API call returns a JSON object that may be mostly metadata. All of it lands back in the context as text, and left unmanaged it is some of the lowest-density content in the entire window — a 5,000-token API response frequently carries fifty tokens of signal wrapped in pagination cursors, null fields, and repeated schema keys.

This makes tool output a prime target for the compression and formatting disciplines, applied at the boundary where the result enters the window. The rule of thumb is: never append a raw tool result verbatim if you can help it. Parse it, select the fields that matter, and inject a compact, labeled version — the three columns you asked for, not the forty the API returned. Where the tool contract is under your control, the highest-value move is to shape the output schema so the tool returns little more than the signal in the first place, rather than returning everything and compressing after. The design of those tool and result schemas — what the model is told it can call, and what shape the answers come back in — is its own discipline, covered in function calling and structured outputs; the context-engineering angle is simply that every tool is a faucet pouring tokens into your window, and unmetered faucets flood it.

There's a second reason to treat tool results carefully, and it isn't about size. Tool output is external content, which means it inherits the trust problem: a search result or a fetched document is authored by someone who is not you, and if it contains text shaped like an instruction, an unguarded pipeline may act on it. Structuring tool output — wrapping it in clear "this is data, not instructions" delimiters — is both a density measure and a safety measure, and the two motivations reinforce each other.

Memory: context that outlives the window

The context window is amnesiac by construction: when a session ends, or when history is trimmed to fit the budget, whatever was in the window is simply gone. Memory is the machinery that fights this — the set of techniques for carrying information across the boundary of a single window so that an assistant feels continuous and a long-horizon agent can work on a task that spans far more steps than fit in one context.

The core idea is externalization. Rather than trying to keep everything in the window (impossible past the token limit) or losing it (unacceptable for anything long-running), you write durable information to storage outside the model and pull the relevant slice back in when needed. Several distinct patterns do this work, and mature systems use more than one:

  • Rolling summarization. As a conversation grows, periodically replace the oldest turns with a running summary — "the user is building a Next.js app, prefers TypeScript, and has already rejected approach A." The recent turns stay verbatim for fidelity; the distant past collapses into a compact recap. This keeps a long session inside the budget while preserving its thread.
  • Compaction. A sharper version of the same move used in agents: when the window approaches its limit, snapshot the essential state — the goal, the decisions made, the facts established, the open questions — into a fresh, dense summary, then discard the raw history entirely and continue from the compacted state. Done well, compaction lets an agent run for hundreds of steps in a window that only ever holds a few dozen steps' worth of material at a time.
  • External long-term memory. Durable facts (preferences, prior decisions, entity details) are stored in a database or vector store keyed for retrieval, and pulled back into the window only when a new query makes them relevant. This is really retrieval pointed at the assistant's own history rather than at a document corpus, and it uses the same chunking and reranking machinery.
  • Structured state files. For agents, the most reliable memory is often the least clever: write intermediate results to a file or a scratchpad and read them back deliberately, so the source of truth lives outside the fragile, lossy context window rather than inside it.

Every one of these is lossy, and that is the point — memory is compression across time, and the engineering judgment is identical to the judgment in the compression section: keep the load-bearing specifics verbatim, summarize the general and the stale aggressively, and accept that you will occasionally drop a detail that turns out to have mattered. The alternative — carrying everything — isn't available past the first few thousand tokens, so the only real question is how gracefully you forget.

Caching: paying for context once

A large, stable context slab has an economic problem: if your system prompt, tool definitions, and few-shot examples run to ten thousand tokens and never change, you are paying to process those same ten thousand tokens on every single call, forever. Prompt (or context) caching is the mechanism that fixes this. The provider stores the processed internal representation of a fixed prefix, so that on subsequent calls with the same prefix it can skip recomputing it and charge a fraction of the normal rate for the cached portion. The savings can be substantial for exactly the workloads context engineering produces — long, stable instruction-and-example preambles followed by a short variable question.

Caching is where context architecture and context economics meet, and it imposes a design constraint that's easy to miss: caching rewards putting your stable content first and your volatile content last. A cache typically matches on a prefix, so if you interleave a changing timestamp or a per-user detail near the top of the window, you invalidate the cache for everything after it and lose the discount. The engineering move is to order the window so the frozen slabs — system prompt, tools, examples — form a long identical prefix across calls, with the volatile slabs (retrieved passages, the user message) appended after the cached boundary. Notice that this pushes in the same direction as the attention argument for ordering: stable, high-priority material up top; volatile material below. When a cost optimization and an accuracy optimization agree, take the free lunch. The exact rates, minimum cacheable sizes, and durations vary by provider and change over time, so treat caching as a real line item to design around, and check the current terms of whichever API you're using rather than assuming; the broader economics are laid out in AI inference cost economics.

The failure modes of a context window

It helps to name the specific ways a context window goes wrong, because "the answer was bad" is a symptom with several distinct diseases, and each has a different fix. Four recur often enough to be worth memorizing.

  • Context rot (lost in the middle). As the window grows long, the model's ability to reliably use any given fact in it degrades — and the degradation is worst for material in the middle. A document you definitely included can be effectively invisible if it's buried at token 60,000 of a 100,000-token context. The tell is a model that "ignores" information you know is present. The fix is upstream: retrieve less, compress more, and place what survives at a high-attention position rather than trusting a long window to surface it.
  • Distraction. Irrelevant-but-plausible content pulls the model off course. The extra retrieved passages that seemed harmless, the tangential section of a document, the stale note in memory — each is a candidate for the model to latch onto instead of the thing that actually answers the question. More context raises the odds of distraction mechanically: every irrelevant token is another thing that might win the model's attention. The fix is relevance density — fewer, better chunks — not more coverage.
  • Poisoning. A false or malicious statement enters the window and the model treats it as ground truth, because within the context everything is equally authoritative. This can be innocent (a stale fact from memory, an outdated retrieved document) or adversarial (injected instructions in a fetched page). Once a poisoned claim is in the window, downstream reasoning inherits it, and in an agent it can propagate for many steps. The fix combines provenance labeling, freshness checks on memory and retrieval, and — for untrusted sources — the defensive posture in the security section below.
  • Conflicting context. Two slabs disagree: the system prompt says one thing and a retrieved document says another; an old summary contradicts a newer fact; two retrieved passages give different figures. The model has no principled way to adjudicate and may pick arbitrarily, average them into something wrong, or hedge into mush. The fix is to resolve conflicts before assembly where you can (dedup, prefer fresh over stale, drop superseded memory) and to make the priority order explicit where you can't ("if the documents conflict with these instructions, follow the instructions").

The unifying observation is that all four get worse as the window gets fuller, which is the empirical heart of "the window is a budget, not a backpack." A bigger context window doesn't cure these failure modes; it just gives them more room to operate. This is also why hallucination and context quality are entangled — a model reaching past thin or contradictory context is a model likely to invent, a dynamic explored in why AI models hallucinate.

Context in agents: the loop that eats itself

Everything above gets harder the moment the system runs on its own. An agent is a loop: it reads its context, decides on an action, calls a tool, and appends the result back into the context — then repeats. Each cycle grows the window. Tool outputs are often verbose (a full API response, a page of search results, a stack trace), and after a dozen steps the context can be mostly the debris of earlier actions.

This is the central engineering problem of agents, and it's why context management, not raw reasoning, is usually what separates an agent that finishes a task from one that spirals. The working techniques mirror the single-shot ones, applied continuously:

  • Summarize completed sub-tasks into a short "here's what I've established" note and drop the raw intermediate steps.
  • Prune tool output to the fields that matter before appending — a 5,000-token API response often carries 50 tokens of signal.
  • Externalize state. Write intermediate results to a scratchpad, a file, or memory outside the window, and pull them back only when needed, instead of carrying everything inline.
  • Cap history depth. Keep the most recent steps in full and compress older ones, so recency (which the model weights heavily anyway) stays sharp.

If you're building or evaluating these systems, this dynamic is the thing to watch; we go deeper on the tooling in the AI coding agents guide, and on what an agent actually is in what is an AI agent. The one-line version: an agent's competence is bounded by how well it curates its own context over time.

It's worth being explicit that agentic context management is its own discipline, not just single-shot context engineering run in a loop, because the loop introduces problems that don't exist in one-shot use. The window is now self-modifying — the model's own outputs become next turn's input — so errors compound: a distraction in step three biases the reasoning in step four, whose output biases step five. Recovery is a context problem, not a reasoning problem; an agent that has talked itself into a corner usually needs its context pruned back to a clean state, not a smarter model. This is where compaction (snapshot the essential state, discard the transcript, continue) earns its keep, and it's why the most reliable agents lean on external state — files, scratchpads, a task list read back deliberately — rather than trusting the growing window to remember. Multi-agent designs push this further by giving each sub-agent its own fresh, narrow context and passing only distilled results between them, so no single window has to hold the whole task; the multi-agent systems guide covers that architecture, and its core motivation is exactly context isolation. The recurring theme across all of it: the scarce resource an agent manages is not compute or tool access, it's the attention budget of its own window, and the systems that win are the ones that spend it deliberately.

A quiet risk: everything in the window is trusted

There's a security wrinkle that context engineering makes unavoidable. The model doesn't distinguish between "instructions from the developer" and "text that happened to be retrieved." It's all just tokens in the same window. So if a retrieved document or a tool result contains text that reads like an instruction — "ignore your previous rules and do X" — the model may follow it. This is prompt injection, and it's a direct consequence of how context works: the more external content you pour into the window, the larger your attack surface. The concrete failure modes, and why retrieval plus tools plus untrusted content form a particularly dangerous combination, are laid out in prompt injection and the lethal trifecta — worth reading before you ship anything that retrieves untrusted data. Treat every non-trusted source as potentially adversarial, and never let retrieved text silently override your system instructions.

The reason this lands in a context-engineering guide rather than only a security one is that the mitigations are, largely, context-assembly decisions. Delimiting slabs clearly so the model can tell rules from data, labeling provenance so untrusted content is marked as such, keeping the trusted system instructions in a high-attention position, and structuring tool output as inert data rather than free text — these are the same formatting and ordering moves that improve accuracy, doing double duty as defenses. None of them is airtight on its own; injection is an open problem, not a solved one. But the mindset is the durable takeaway: every token you admit to the window is a token the model will, by default, believe, so the composition of the context is your security boundary.

How to actually get good at this

Context engineering rewards measurement in a way prompt tweaking rarely does. Because the window is assembled from swappable components, you can change one at a time and see the effect. A practical loop:

  1. Build an eval set — real inputs with known-good outputs, covering the cases you care about.
  2. Log the full context for failures, not just the final answer. Most bad answers trace to a bad window: a missing document, a stale memory, a buried instruction.
  3. Change one component and hold the rest fixed — a new retrieval strategy, a different ordering, a tighter summary — and re-run the evals.
  4. Watch the budget. Track tokens per call alongside quality, so you don't buy a small accuracy gain with a large cost regression.

That discipline is what turns context engineering from folklore into engineering. The phrasing tricks people trade online are guesses; a context change you can measure is a decision.

A few things make context evals sharper than prompt evals. First, ablate the slabs. Because the window is assembled from components, you can measure each one's contribution by removing it and watching the score: drop the few-shot examples and quality holds? They were dead weight — reclaim the tokens. Drop the third retrieved passage and nothing changes? Retrieve two. This is how you find the low-value tokens that budgeting demands you cut, empirically rather than by intuition. Second, test at the edges of the budget, not just typical cases: the failure that matters is often the unusually long conversation or the oversized tool result that pushes a normally-fine context past the point where lost-in-the-middle sets in, and that only shows up if your eval set includes long, cluttered inputs. Third, separate retrieval quality from generation quality. If answers are wrong, the first question is whether the right facts were even in the window; measuring retrieval hit-rate independently of final-answer accuracy tells you whether to fix the retriever or the prompt, and it's astonishing how often teams tune the wording when the real problem is that the evidence never arrived. For agentic systems the same logic extends across the whole trajectory rather than a single call — the discipline of scoring multi-step runs is its own topic, covered in how to evaluate AI agents. The through-line: context engineering is falsifiable, and treating it that way is the entire difference between a system that improves and one that just changes.

FAQ

What is context engineering in simple terms? Context engineering is the practice of deciding exactly what information a model sees when it generates a response — the instructions, retrieved facts, conversation history, tool results, and memory that all share one fixed-size context window — and how to order and compress that information so the model can use it well. It's the systems-level successor to writing a single good prompt.

How is context engineering different from prompt engineering? Prompt engineering optimizes the wording of one request. Context engineering optimizes the entire input the model reads, most of which isn't the prompt at all: retrieved documents, history, tool outputs, and memory. Prompting is about phrasing; context engineering is about assembly, ordering, and budget. A great prompt in a badly-built context still fails.

Does a bigger context window make context engineering unnecessary? No. Larger windows raise the ceiling but don't remove the problem. Every token still costs money and latency, models attend unevenly across long inputs (material in the middle gets underweighted), and more text means more noise competing with your signal. A bigger window makes over-stuffing easier, not smarter. The goal is the smallest context that fully supports a correct answer.

Why does the order of information in the context matter? Because models don't read a long input evenly. Content near the beginning and end of the window tends to influence the output more than content buried in the middle — often called the "lost in the middle" effect. So placing your most important instruction or most relevant retrieved passage at a high-attention position can change the answer, even with identical content.

Is RAG the same as context engineering? No — RAG is one tool within context engineering. Retrieval-augmented generation handles getting relevant external facts into the window, but context engineering also covers compression, ordering, conversation history, long-term memory, tool output management, and token budgeting. RAG fills part of the context; context engineering governs the whole thing.

Why is context management the hard part of building agents? Because an agent runs in a loop that keeps appending tool results to its own context, the window grows every step and quickly fills with the debris of earlier actions. Without active management — summarizing finished sub-tasks, pruning verbose outputs, externalizing state — the agent runs out of budget, loses the thread, or drowns its own reasoning in noise. Curating that growing context is the core engineering challenge.

What is context rot, and how do I prevent it? Context rot is the empirical decline in a model's ability to reliably use information as the window gets longer — worst for material in the middle of a long input, where a fact you definitely included can become effectively invisible. You don't prevent it by trimming the window after the fact; you prevent it upstream, by retrieving fewer and better passages, compressing aggressively, and placing the most important material at the high-attention start and end of the context rather than trusting a long window to surface it. The practical rule: if the model is "ignoring" something you know is present, the fix is a shorter, better-ordered window, not a firmer instruction.

Does prompt caching change how I should order my context? Yes, and it happens to agree with what accuracy already wants. Caching stores the processed form of a fixed prefix so you don't pay to recompute it every call, but it typically matches on an exact prefix — so any volatile detail (a timestamp, a per-user value, the retrieved passages) placed early invalidates the cache for everything after it. The move is to front-load the stable slabs (system prompt, tool definitions, few-shot examples) as a long identical prefix and append the volatile slabs after the cache boundary. That's the same ordering the attention argument recommends — stable and high-priority up top, volatile below — so the cost win and the quality win point the same way.

How much context is too much? There's no fixed token number; the right test is marginal value. For each slab, ask whether the last chunk you added measurably improved the output on your eval set — the fifth retrieved passage, the twelfth example, the twentieth turn of history. If removing it doesn't hurt quality, it was too much, and it was costing you money, latency, and attention the whole time. "Too much" is reached well before the window is full: the goal is the smallest context that fully supports a correct answer, and the only reliable way to find that point is to ablate components and watch the scores, not to fill the window because the capacity exists.