Prompt20
NewsDataBlog
All posts
llm-as-a-judgeevaluationevalrubricsbiastestinghow-toevergreen

LLM-as-a-Judge: Using AI to Evaluate AI (Reliably)

How to use a model to grade outputs at scale, where judges are biased — position, verbosity, self-preference — and how to design rubrics and calibrate against humans so the scores mean something.

By Prompt20 Editorial · 24 min read

LLM-as-a-judge is the practice of using one language model to grade the outputs of another — or of itself — against a rubric, instead of hiring humans to read every response. It works because judging is easier than generating: a model that can't reliably write a flawless answer can often reliably tell a good answer from a bad one, the same way a mediocre writer can be a sharp editor. That asymmetry is the whole business case. It lets you score thousands of outputs for the price of a few API calls and get a number back in minutes instead of a week.

The catch, and the reason this article exists, is that a judge is just another model with another prompt, and it fails in specific, repeatable ways. It prefers longer answers. It prefers whatever appears first. It prefers text that sounds like its own. If you deploy an LLM judge without measuring those biases, you don't have an evaluation — you have a random number generator wearing a lab coat. The good news is that every one of these failure modes is known, and most are cheap to defend against once you know they exist.

Key takeaways

  • A judge is a model, not an oracle. Its scores are opinions produced by a prompt, and they inherit that prompt's blind spots. Treat the judge as a system to be tested, not a source of ground truth.
  • The big biases are named and predictable: position bias, verbosity bias, self-preference, and sycophancy. You can measure and largely neutralize all four.
  • Pairwise comparison beats absolute scoring for reliability. "Is A or B better?" is a question models answer far more consistently than "Rate this 1–10."
  • Calibrate against humans before you trust a number. If your judge doesn't agree with human graders on a labeled sample, the automated scores are decoration.
  • A good rubric does most of the work. Vague criteria produce vague, biased judgments; concrete, checkable criteria produce judgments you can defend.

Table of contents

Why judge with a model at all

Human evaluation is the gold standard and it does not scale. A careful human rater might grade a few hundred model responses a day, costs real money, gets tired, and disagrees with the next human rater more often than anyone likes to admit. When you're iterating on a prompt or a fine-tune, you need to re-grade the whole test set every time you change something. At that cadence, human-only evaluation becomes the bottleneck that kills your iteration speed.

Automated string-matching metrics — exact match, BLEU, ROUGE and their descendants — scale fine but only measure surface overlap with a reference answer. They can't tell you whether a summary is faithful, whether an answer is helpful, or whether a chatbot's tone is appropriate. For open-ended generation, they are close to useless.

LLM-as-a-judge sits in the gap. It reads the actual content the way a human would, applies criteria you specify in natural language, and returns a verdict at machine speed and machine cost. That makes it the practical backbone of most modern eval pipelines: fast enough to run on every commit, flexible enough to grade open-ended text, cheap enough to run at scale. It is not a replacement for human judgment. It is a force multiplier on top of a small amount of human judgment — and it only works if you keep that human anchor in the loop.

The three jobs a judge does

Not all judging is the same task. There are three common patterns, and choosing the right one matters more than choosing the model.

Mode Question asked Best for Main weakness
Pairwise comparison "Is response A or B better?" Comparing two models, prompts, or versions Doesn't give an absolute quality level
Single-answer grading "Score this response 1–5 on X" Tracking quality over time; large sets Scores drift and cluster; hard to calibrate
Reference-based grading "Does this match the correct answer?" Tasks with a known right answer Needs a trusted reference for every item

Pairwise is the most reliable because relative judgments are more stable than absolute ones — a model asked to pick the better of two answers is far more consistent than the same model asked to assign a number on a scale it has to invent. Single-answer grading is the most convenient because you get a standalone score you can average and chart, but those scores tend to bunch up (everything gets a 4) and shift between runs. Reference-based grading is the most objective when you have references, and irrelevant when you don't.

A practical pattern: use pairwise comparisons to decide "did this change help?" during development, and use single-answer grading with a tight rubric for ongoing quality dashboards. Don't average scores from different rubrics or different judge versions and pretend they're the same metric.

Why relative beats absolute, mechanically. When you ask a model to rate an answer 7 out of 10, it has to hallucinate a reference frame: seven relative to what distribution of answers, anchored where? Different runs invent different anchors, so the numbers wobble even when the model's underlying opinion is stable. A pairwise question removes the anchor problem entirely — the model only has to decide which of two concrete artifacts it prefers, a judgment humans and models both make far more reliably than absolute rating. This is the same reason product teams A/B test rather than asking users to score a design 1–10 in isolation.

The cost of pairwise is that a single comparison tells you "B beat A" but not "by how much" or "compared to everything else." You recover a global ranking by running many comparisons and aggregating them, exactly the way chess ratings work. Feed pairwise win/loss records into an Elo or Bradley–Terry model and you get a stable, interpretable leaderboard where each candidate has a rating and the gaps between ratings mean something. This is how public model-comparison arenas turn millions of noisy individual votes into a rank order — and you can run the same machinery internally on a few hundred judge comparisons to rank prompt variants or checkpoints. The practical upshot: if you have more than two things to compare, don't score each one absolutely; run a round-robin (or a smart subset) of pairwise matches and let the aggregation produce the ranking.

Reference-based grading hides two very different setups. In the strict version the reference is the answer and the judge checks equivalence — useful for math, code with known outputs, or closed factual questions. In the loose version the reference is an exemplar of a good answer, and the judge grades similarity of quality rather than identity of content — useful for summaries where many phrasings are equally right. Confusing the two is a common failure: grading an open-ended answer as if it had to match a golden string punishes good answers for being worded differently, and your scores track surface overlap instead of quality. If your task has many valid answers, use the exemplar as a rubric anchor, not a string to match.

Scoring mechanics: chain-of-thought, structured output, and probabilities

How you extract the verdict from the model matters almost as much as what you ask. Three mechanics account for most of the difference between a flaky judge and a dependable one.

Reason first, then commit. A judge asked to emit a bare score computes that score in a single forward pass with no room to work through the evidence — it pattern-matches to a number. Requiring a short written rationale before the verdict forces the model to surface the specific facts it is grading on, and the score conditioned on that rationale is measurably more grounded. This is not mysticism; it is the same reason chain-of-thought helps on reasoning tasks. The rationale also gives you an audit trail: when a score looks wrong, you can read why the judge decided it, and often discover the judge misread the task rather than the answer being bad. The one discipline that matters here is ordering — reasoning must come before the score in the output, because if the score is emitted first the model rationalizes a number it already committed to.

Constrain the output shape. Free-form judge responses are a parsing nightmare and a reliability leak: the model buries the verdict in prose, hedges, or answers a slightly different question. Force structured output — a JSON object with named fields for each rubric criterion, each constrained to your allowed values ("pass"/"fail", or an integer in a fixed range). Structured output does two jobs at once: it makes the result machine-readable without brittle regex, and it constrains the model to actually answer every criterion you asked about instead of the one it found easiest. Put the free-text rationale in its own field so you keep the reasoning without letting it contaminate the parse.

Use the token probabilities when you can. A model that outputs "pass" is also, under the hood, assigning a probability to "pass" versus "fail." If your setup exposes token log-probabilities, you can read that probability and get a graded confidence signal instead of a hard binary — a judge that is 51% confident and one that is 99% confident both print "pass," but only the probability distinguishes them. Aggregating these soft scores across a test set is smoother and more sensitive to small quality changes than counting hard verdicts, which is exactly what you want when you are trying to detect whether a change moved quality by a little. When log-probs are unavailable, the cheap approximation is to sample the judge a handful of times at nonzero temperature and use the fraction of "pass" votes as the confidence — more on that under ensembling.

The four biases that ruin judges

These are the failure modes that turn a judge's output into noise. Each has a defense.

Position bias

Ask a model "is A or B better?" and it has a measurable tendency to prefer whichever answer is in a particular slot — often the first one — regardless of content. Swap the two answers and the verdict can flip. This is not subtle; on some setups it's large enough to swamp real quality differences.

Defense: run every pairwise comparison twice with the order swapped, and only count a win if the same answer wins both times. Ties (A wins in one order, B in the other) get flagged as "no consistent preference," which is itself useful information. Yes, it doubles your judging cost. It's still cheaper than shipping a conclusion that was an artifact of ordering.

Verbosity bias

Judges reward length. Given two answers of equal correctness, models tend to score the longer, more elaborate one higher — more headers, more caveats, more restating of the question. This is dangerous precisely because it's plausible-looking: the verbose answer feels more thorough. Left unchecked, verbosity bias will quietly train your whole system to be windier, because any optimization loop that uses the judge as its reward will discover that padding raises the score.

Defense: make concision an explicit rubric criterion, or normalize for length by including short high-quality answers in your calibration set and checking that they aren't being penalized. If you're using the judge as a reward signal for training or prompt optimization, treat verbosity as a reward-hacking risk and monitor output length as a metric in its own right. This is the same dynamic that shows up whenever a proxy metric gets optimized — see benchmark hacking and reward hacking for the general pattern.

Self-preference bias

Models tend to rate text generated by themselves — or by models in their own family, with their own stylistic fingerprints — more highly than text from other sources. If you use Model X as the judge in a bake-off that includes Model X as a contestant, you've built a conflict of interest into your evaluation.

Defense: don't let a model judge its own outputs in a competitive eval. Use a different model family as the judge, or at minimum disclose and discount the result. When comparing several models, a judge from outside the candidate pool is the cleanest choice.

Sycophancy and framing effects

Judges are swayed by how the question is framed. Tell the judge "this answer was written by an expert" and scores rise. Include the author's confident self-assessment in the text ("This is clearly the correct approach") and the judge tends to go along. Ask a leading question and you get the answer you led toward.

Defense: strip identifying and self-promoting framing from the material before it reaches the judge. Keep the judge prompt neutral — "evaluate the following" beats "evaluate this excellent answer." Never tell the judge which answer you hope wins.

More failure modes: the biases nobody warns you about

The famous four get the headlines, but a production judge runs into a longer tail of quieter distortions. None of these are exotic; they are just less discussed, which makes them more dangerous because nobody checks for them.

Formatting and authority bias. Judges reward the appearance of rigor. Bullet points, bold headers, numbered steps, a confident tone, and citations — even fabricated ones — nudge scores up independent of substance. An answer that says "According to research, X" tends to beat a plain "X" even when neither is sourced and both are equally correct. This is verbosity bias's better-dressed cousin, and it is why a system optimized against a naive judge drifts toward answers that look like a consulting deck. Defense: put "is every claim actually supported?" and "does the format serve the reader or just decorate?" in the rubric, and include a well-formatted-but-wrong answer in your calibration set to confirm the judge isn't fooled by it.

Anchoring on the first criterion. When a rubric lists several criteria, the judge's read of the first one colors the rest — an answer marked strong on criterion one gets a benefit of the doubt on criteria two through five. Defense: score each criterion in its own isolated call when reliability matters more than cost, or at minimum require the judge to justify each criterion independently rather than emitting one holistic impression split across fields.

Distribution and scale compression. On a 1–5 scale, real judges almost never use 1 or 5; everything piles into 3 and 4. This compresses the very signal you are trying to measure — a quality regression that should move a score from 4.2 to 3.6 gets muffled into 4.1 to 3.9. Defense: prefer binary criteria that you sum into a score (six yes/no checks give you a 0–6 that actually spreads out) over a single Likert rating the model refuses to use fully.

Content and identity bias. Judges carry the same demographic, political, and stylistic leanings as the models they are built from. In a subjective task they may systematically favor answers that match their training distribution's preferences — a real fairness problem if you are grading anything where viewpoint or dialect legitimately varies. Defense: for subjective or sensitive tasks, audit judge verdicts across the axes you care about, and lean harder on human review; some judgments should not be automated at all (see when not to use an LLM judge).

Refusal and safety over-triggering. A judge asked to evaluate content near a safety boundary sometimes refuses to grade it at all, or marks a perfectly good answer down because the topic is sensitive. Your eval then silently drops or penalizes exactly the hard cases you most need measured. Defense: monitor the judge's refusal rate as a metric, and route refused items to humans rather than scoring them as failures.

Designing a rubric that survives contact with reality

Most bad LLM-judge setups fail at the rubric, not the model. "Rate the helpfulness of this answer from 1 to 10" is not a rubric; it's an invitation for the model to invent a scale on the spot and apply it inconsistently. A rubric is a set of concrete, checkable criteria that a careful stranger could apply and reach roughly the same conclusion you would.

Principles that hold up:

  • Decompose the judgment. Instead of one fuzzy "quality" score, ask for several specific ones: Is it factually correct? Does it answer the actual question? Is it appropriately concise? Does it follow the format requested? Specific questions get consistent answers; global ones get vibes.
  • Prefer binary or low-cardinality scales. "Yes / no / partially" is more reliable than 1–10. Models can't meaningfully distinguish a 7 from an 8, and neither can most humans. Fine-grained scales manufacture false precision.
  • Give the judge an escape hatch. Include "cannot determine from the given information" as an option. Without it, the judge will guess, and guesses look identical to knowledge in the output.
  • Ask for a reason before the verdict. Requiring the judge to write its reasoning first, then commit to a score, produces more grounded judgments than asking for the score alone — and gives you an audit trail when you want to know why something was marked down. The reasoning also lets you spot when the judge misread the task entirely.
  • Show, don't just tell. A couple of worked examples — "here's a 'yes', here's a 'no', here's why" — anchor the judge far better than adjectives. This is ordinary prompt engineering, and the same skills apply; if that's unfamiliar, how to write better prompts covers the fundamentals.

The test of a rubric is simple: hand it to two different people (or two runs of the judge) and see if they agree. If they don't, the rubric is underspecified, and no amount of model quality will fix it.

Calibration: the step everyone skips

Here is the part that separates real evaluation from theater. Before you trust a judge's scores, you have to prove the judge agrees with humans on a sample where you know the truth.

The process is unglamorous and non-negotiable:

  1. Build a small labeled set. Have humans carefully grade 50–200 representative outputs using the same rubric the judge will use. This is your ground truth. Include hard cases and edge cases, not just easy ones.
  2. Run the judge on the same set. Now you have two columns: human verdicts and judge verdicts.
  3. Measure agreement. Compute how often they match. For pairwise, that's agreement rate; for scores, look at correlation and at the confusion pattern — where does the judge disagree, and in which direction?
  4. Decide if it's good enough. A useful sanity check: how often do two humans agree with each other on this task? If your judge agrees with humans about as often as humans agree with each other, the judge is as reliable as a human rater — which is all you can ask. If human-human agreement is low, the task itself is ambiguous and no judge will save you.
  5. Re-check when anything changes. New judge model, new rubric wording, new domain of inputs — any of these can break calibration. Treat the labeled set as a regression test you re-run, not a one-time ceremony.

The reason this matters: a judge can be precise (gives the same answer every time) without being accurate (gives the right answer). Consistency is easy and seductive. A judge that reliably marks the wrong answer as correct will produce beautiful, stable, completely misleading dashboards. Calibration against humans is the only thing that catches this. Without it, you are measuring your judge's opinions, not your system's quality.

Calibration metrics: agreement, kappa, and correlation done right

"Measure agreement" sounds simple until you try to pick a number, and the naive choice — raw percent agreement — is quietly misleading. Here is what to actually compute and how to read it.

Raw agreement, and why it flatters. Raw agreement is just the fraction of items where judge and human give the same verdict. It is easy and worth reporting, but it overstates reliability whenever the classes are imbalanced. If 90% of your answers are "pass," a judge that blindly says "pass" every time scores 90% agreement while having learned nothing. Any time one verdict dominates, raw agreement is inflated by luck and you need a chance-corrected metric.

Cohen's kappa: agreement above chance. Kappa answers the question raw agreement dodges: how much do the two raters agree beyond what random guessing at the same base rates would produce? It subtracts the agreement you'd expect by chance and rescales, so 0 means "no better than coin-flipping with the right bias" and 1 means "perfect." A judge with 90% raw agreement but near-zero kappa is a "pass" rubber stamp, not an evaluator. There is no universal threshold that means "good" — the honest interpretation depends on your task's difficulty — but the pattern to internalize is directional: compare the judge's kappa against your human-human kappa on the same items. If two humans only reach moderate agreement, expecting the judge to reach near-perfect agreement is incoherent; the ceiling is set by how well-defined the task is, not by the judge.

Correlation for graded scores. When the judge emits a number rather than a category, agreement becomes correlation between judge scores and human scores across the set. Rank correlation (does the judge order items the same way humans do?) is usually what you care about, because for most decisions you want to know whether the judge can tell better from worse, not whether it lands on the identical integer. A judge can be biased high on every item — always two points generous — yet still rank items perfectly, and a high rank correlation tells you that judge is fine for comparisons even though its absolute numbers need offsetting.

Read the confusion pattern, not just the score. A single agreement number hides the most useful information: where and which direction the judge disagrees. Build the confusion table — judge-pass/human-fail and judge-fail/human-pass are different diseases with different cures. Systematic false-positives (judge too lenient) mean your dashboards will overstate quality and hide regressions; systematic false-negatives (judge too harsh) mean you'll chase phantom problems. A judge that disagrees symmetrically and rarely is trustworthy; one that disagrees in a consistent direction is biased and you can often correct it by tightening the rubric on exactly the criterion where the disagreements cluster.

The whole point of these metrics is to convert a vague feeling ("the judge seems decent") into a defensible statement ("the judge agrees with humans as well as humans agree with each other, with no directional bias"). Until you can make that statement, treat the judge's scores as a hypothesis, not a measurement.

Ensembling judges and taming variance

A single judge call is a single sample from a distribution. At temperature zero it's more stable but still brittle to prompt phrasing; at nonzero temperature it visibly varies run to run. When a quality difference you care about is smaller than the judge's own noise, you can't see it. Ensembling is how you shrink that noise.

Self-consistency: sample the same judge several times. Run the same judge on the same item three, five, or seven times and take the majority verdict (or the mean score). Because the errors are partly random, averaging cancels them, and the aggregate is more reliable than any single call — the same self-consistency trick that improves reasoning tasks. The fraction of votes for "pass" also doubles as a free confidence estimate: a 5–0 sweep is a different signal than a 3–2 squeaker, and you can route the close calls to a human. The cost is linear in the number of samples, so reserve heavy sampling for the decisions that matter (a release gate) and use single calls for cheap continuous monitoring.

Panels: different judges, different blind spots. A single model family has correlated errors — sampling it more just gives you the same bias more precisely. A panel of different model families as judges gives you partly independent errors, which is what actually reduces systematic bias rather than variance. When the panel agrees, you can be confident; when it splits, you've found a genuinely ambiguous case worth human eyes. A panel also directly defuses self-preference bias, because no single model's stylistic favoritism can dominate a mixed jury. The trade-off is cost and coordination: a three-model panel triples spend and forces you to define how disagreements resolve (majority vote, or escalate ties to a human).

Know when ensembling can't help. Ensembling reduces variance — random scatter. It does nothing for bias — a systematic error shared by every judge in the ensemble. If every model you sample rewards verbosity, sampling ten of them just gives you a very stable preference for long answers. Only calibration against humans catches bias; ensembling only makes an already-calibrated judge quieter. Do the calibration first, then ensemble to tighten the signal.

Reward-hacking the judge

The moment a judge stops merely reporting quality and starts driving an optimization — a training reward, a prompt-tuning loop, a "keep generating until the judge says 8+" filter — its biases stop being measurement error and become an attack surface. Any optimizer pointed at a proxy will find and exploit the gap between the proxy and the thing you actually wanted. The judge is a proxy. This is Goodhart's law with a language model in the loop: when the judge's score becomes the target, it stops being a good measure.

The mechanism is unglamorous and reliable. If the judge rewards length, the optimized system gets longer. If it rewards confident tone, outputs get more confident regardless of correctness. If it rewards citations, the system learns to emit citation-shaped strings — including fabricated ones, because the judge rewards the form of a citation, not its truth. If it rewards a particular structure, everything becomes bullet points. None of this requires the model to "cheat" in any intentional sense; gradient descent and search simply flow downhill toward whatever the judge scores highest, and every bias you didn't close is a downhill direction.

Defenses, in order of importance. First, hold out a judge the optimizer never sees. If you optimize against Judge A, evaluate final quality with a different Judge B (ideally a different model family) that the optimization loop had no access to — a gap between the two is your reward-hacking alarm. Second, monitor the tells directly. Track output length, citation count, refusal rate, and format density as first-class metrics; when the judge's score climbs while these balloon, you are watching hacking happen, not quality improving. Third, keep a human sample in every optimization cycle, not just at the start — the whole risk is that the system drifts away from what humans want while the judge keeps applauding, and only fresh human labels catch that drift. Fourth, prefer harder-to-hack judge designs: pairwise against a fixed strong reference is more robust than an absolute score the optimizer can inflate, and criterion-decomposed rubrics give the optimizer fewer single knobs to crank. The general pattern — proxy metrics degrade under optimization pressure — is the same one that breaks benchmarks, covered in benchmark hacking and agent reward hacking.

Judges in CI: regression eval that actually gates

The highest-leverage use of an LLM judge is turning evaluation into a test suite — something that runs on every change to your prompt, model, or pipeline and tells you, before you ship, whether quality moved. Done right, this is the difference between "we think the new prompt is better" and "the eval says helpfulness held and format-adherence went up two points." Done wrong, it's a flaky red X everyone learns to ignore.

The core tension is that unit tests are deterministic and judge-based evals are not. A conventional test passes or fails identically every run; a judge, especially at nonzero temperature, can flip on a borderline case and turn your CI red for no real reason. If you gate merges on a metric that jitters, the team stops trusting it, which is worse than having no gate. The fixes are practical. Pin the judge to temperature zero and a frozen, versioned judge prompt so the grader itself doesn't drift under you — treat the judge prompt as code, reviewed and version-controlled, because changing its wording silently rescopes every historical comparison. Gate on aggregate metrics over a fixed dataset, not per-item verdicts: "average faithfulness across 200 cases dropped more than X" is a stable signal; "case 47 flipped" is noise. Set the threshold using the judge's own measured noise floor — run the identical system through the eval a few times, see how much the aggregate wanders on no change at all, and set the alert band wider than that so you flag real regressions, not jitter.

Structure the dataset like a test suite, not a random sample. Keep a golden set of representative cases for the headline number, and a growing regression set where every past bug becomes a permanent case — when a specific failure gets reported and fixed, you add an example of it so the eval will scream if it ever comes back. That regression set is where the compounding value lives; it encodes everything the system has ever gotten wrong. Run the deterministic and cheap checks first (schema, length, forbidden phrases) and only spend judge calls on cases that pass those, so CI stays fast and cheap. The infrastructure to store datasets, version judge prompts, and track these scores over time is its own subject — the eval infrastructure walkthrough covers the harness side, and for multi-step agents where a "case" is a whole trajectory rather than one response, the agent evaluation guide handles the extra machinery.

Cost and reliability trade-offs vs human and code graders

An LLM judge is one of three graders you can point at any output, and choosing well means being honest about what each buys you. Code graders — exact match, schema validation, regex, unit tests on generated code — are effectively free, instant, perfectly deterministic, and completely reliable within their reach, which is narrow: they can only check things with a crisp mechanical definition. Human graders are the gold standard for judgment and the only true source of ground truth, but they are slow, expensive, inconsistent between raters, and impossible to run on every commit. The LLM judge sits deliberately in between: far cheaper and faster than humans, far more flexible than code, and less reliable than either within their respective strengths.

The engineering move is to use the cheapest grader that can actually answer the question, and never to pay for a more expensive one out of habit. Spending a judge call to check whether output is valid JSON is waste — a parser is free and correct. Spending a human on 10,000 tone judgments is waste — a calibrated judge does it for cents. Spending a judge on a high-stakes subjective call that humans themselves barely agree on is a false economy — you get a cheap number that means nothing. The layering that falls out of this is: code for the mechanical, judge for the open-ended-but-checkable, humans for the ambiguous, the high-stakes, and the sample you use to keep the judge honest.

Two costs of the judge are easy to underestimate. First, the judge is not free to build even though it's cheap to run — the calibration work, the labeled set, the rubric iteration, and the ongoing re-calibration are recurring human effort, and a judge you built once and never re-checked is a liability dressed as an asset. Second, judge cost scales with your defenses: order-swapping doubles pairwise cost, self-consistency multiplies by sample count, panels multiply by judge count, per-criterion isolation multiplies by criteria. A fully defended judge can cost 10–20x a naive single call — still trivial next to human grading, but enough that "use a bigger judge model on every check" is rarely the right default. Reserve the expensive, heavily-defended configuration for release gates; use lean single calls for continuous monitoring.

When not to use an LLM judge

The technique has a boundary, and pretending it doesn't is how teams end up with confident, meaningless dashboards. Skip the LLM judge when:

  • A code grader can decide it. Anything with a crisp definition — valid JSON, exact numeric answer, passing unit tests, presence of a required field — should be checked deterministically. The judge adds cost, latency, and new failure modes to a question that has an exact answer. Never launder a decidable question through a probabilistic grader.
  • Humans can't agree on the answer. If your ground-truth labels themselves disagree wildly — deeply subjective taste, contested values, genuinely open questions — no judge can be more reliable than the humans it's calibrated against, and its confident single verdict papers over a real disagreement you should surface, not hide. Low human-human agreement is a signal to redefine the task or keep humans in the loop, not to automate it.
  • The stakes are too high for the reliability you can prove. For a decision where a wrong grade causes real harm — medical, legal, safety-critical, anything irreversible — the judge's accuracy has to clear a bar you can rarely demonstrate with a calibration set. Use it as a triage filter that flags items for humans, never as the final authority.
  • The judge is inside the loop it's grading. A model grading its own outputs in a competitive comparison, or the exact model being optimized serving as its own reward signal without an independent check, builds the conflict of interest directly into the number. Get an outside judge or don't trust the result.
  • You haven't calibrated yet. Before any human-labeled agreement measurement exists, you don't have an evaluation — you have vibes with a decimal point. That's fine for exploration; it is not fine for any decision, and it must never be reported as if it were measured.

The honest framing: the LLM judge is a scaling tool for judgments humans could make but can't make often enough. Where humans couldn't reliably make the judgment either, or where a machine can make it exactly, the judge is the wrong tool.

A practical setup, end to end

Pulling the threads together, here is a concrete recipe that avoids the common traps. Treat it as a starting skeleton, not gospel — the calibration step is what tells you where your task deviates from it.

  1. Write the rubric as decomposed, low-cardinality criteria. Three to six specific yes/no or three-way questions ("Is every factual claim supported? Does it answer the question asked? Is it within the requested format? Is it appropriately concise?"), each with a one-line definition and a worked example of pass and fail. Sum the binary criteria into a score that actually spreads out.
  2. Fix the judge prompt and freeze it. Neutral framing, reasoning-before-verdict, structured JSON output with one field per criterion plus a rationale field, temperature zero, and a version number. Check it into version control alongside your code and never edit it silently.
  3. Choose the mode for the job. Pairwise against a fixed reference for "did this change help?"; summed-binary single-answer grading for the ongoing dashboard. Use a judge from a different model family than any system it's grading competitively.
  4. Build the labeled calibration set. 50–200 representative items, deliberately including hard and edge cases, graded by humans with the exact rubric. Have two humans grade an overlap so you know the human-human agreement ceiling.
  5. Calibrate. Run the judge on the set; compute chance-corrected agreement (kappa) or rank correlation; read the confusion pattern for directional bias. If the judge trails human-human agreement badly or leans consistently one way, fix the rubric on the criterion where disagreements cluster and repeat. Do not proceed on raw percent agreement alone.
  6. Defend by construction. Swap answer order on every pairwise call and require a consistent winner; strip identifying and self-promoting framing from inputs; monitor length, citation count, refusal rate, and format density as their own metrics.
  7. Wire it into CI as a gate on aggregate metrics. Fixed golden set for the headline number, a regression set that grows one case per fixed bug, thresholds set wider than the judge's measured no-change noise floor, cheap deterministic checks running first.
  8. Ensemble only where it pays. Single calls for continuous monitoring; self-consistency sampling or a multi-model panel for release gates and close calls; route low-confidence verdicts to humans.
  9. Re-calibrate on every change. New judge model, new rubric wording, new input domain — re-run the labeled set. Refresh the human sample periodically even when nothing changed, because your input distribution drifts even when your code doesn't.

Where LLM judges fit in a pipeline

LLM-as-a-judge is one component, not the whole evaluation strategy. It slots in alongside cheaper deterministic checks and scarcer human review. A sensible layering:

  • Deterministic checks first. Did the output parse as valid JSON? Is it within the length limit? Does it contain a forbidden phrase? These are free, instant, and don't need a model. Never spend a judge call on something a regex can decide.
  • LLM judge for the open-ended middle. Faithfulness, helpfulness, tone, adherence to nuanced instructions — the stuff only a reader can assess. This is the judge's home turf.
  • Humans on the samples that matter. Spot-check the judge's verdicts, adjudicate disagreements, and re-label periodically to keep calibration honest. Humans are the anchor the whole system hangs from, used sparingly.

This mirrors how larger eval systems are built; for the infrastructure view — datasets, harnesses, tracking scores over time — see the eval infrastructure walkthrough, and for the specific challenges of grading multi-step agents rather than single responses, the agent evaluation guide goes deeper. The judge is a reusable building block that shows up across all of these.

FAQ

Is LLM-as-a-judge accurate enough to trust? It's accurate enough to be useful when calibrated against human judgment on your specific task, and untrustworthy when deployed blind. A judge's raw agreement with humans varies enormously by task — high on clear-cut factual grading, lower on subjective quality. The honest answer is: measure agreement on a labeled sample first, and only trust the judge as far as that measurement justifies. Never assume; always verify against humans.

Can a model reliably judge its own outputs? Not in a competitive setting. Models exhibit self-preference bias — they rate their own text and text from their own model family more favorably. For grading a single system's outputs against a fixed rubric this is a smaller concern, but any time a model is both a contestant and the referee, use a judge from a different model family to remove the conflict of interest.

Should I use pairwise comparison or absolute scoring? Use pairwise ("is A better than B?") when comparing two versions, models, or prompts — it's more reliable because relative judgments are more stable than absolute ones. Use single-answer scoring when you need a standalone quality number to track over time, but expect scores to cluster and drift, and always run comparisons with answer order swapped to cancel out position bias.

How do I stop the judge from just preferring longer answers? Make concision an explicit rubric criterion rather than hoping the judge ignores length, and include short high-quality answers in your calibration set to confirm they aren't penalized. If the judge feeds a training or optimization loop, treat verbosity as a reward-hacking risk and monitor output length as its own metric, because any optimizer will exploit a length-biased reward.

How many human labels do I actually need to calibrate? Fewer than people fear — often 50 to 200 carefully graded, representative examples are enough to estimate judge-human agreement and catch systematic errors. The value is in quality and coverage of hard cases, not raw quantity. Re-label a fresh sample whenever you change the judge model, the rubric, or the domain of inputs, since any of those can silently break calibration.

Does a bigger or newer judge model give better evaluations? Not automatically. A stronger model can follow a rubric more faithfully, but it carries the same structural biases — position, verbosity, self-preference — and a better model with a vague rubric still produces vague judgments. The rubric and the calibration process drive reliability far more than the choice of judge model. Fix those before reaching for a bigger model.

Should I make the judge explain its reasoning before scoring? Yes, and the order is the whole point. Requiring a short rationale before the verdict forces the model to surface the specific evidence it's grading on, and a score conditioned on that reasoning is more grounded than a bare number — the same effect chain-of-thought has on reasoning tasks. The rationale is also your audit trail: when a score looks wrong, you read why the judge decided it, and you'll often find it misread the task rather than the answer being bad. Just never let the score come first, or the model rationalizes a number it already committed to.

How do I keep a judge stable enough to gate merges in CI? Freeze it: temperature zero, a versioned judge prompt treated as reviewed code, and gating on aggregate metrics over a fixed dataset rather than per-item verdicts that flip on borderline cases. Set the alert threshold wider than the judge's own measured no-change noise floor — run the same system through the eval a few times and see how much the aggregate wanders on zero changes, then flag only movements larger than that. A regression set that gains one case per fixed bug is where the compounding value lives.

Can I use one judge to optimize a system and also to prove it improved? No — that's how reward-hacking hides. If an optimizer (training, prompt-tuning, best-of-N filtering) is pointed at Judge A, evaluate the final result with a held-out Judge B from a different model family that the loop never touched. A gap between the two is your hacking alarm. Also monitor the tells directly — length, citation count, format density, refusal rate — because a score that climbs while those balloon is the optimizer exploiting the judge, not quality improving.

The bottom line

LLM-as-a-judge is one of the highest-leverage techniques in applied AI evaluation, and one of the easiest to do badly. The leverage comes from the generation-versus-judging asymmetry: grading is genuinely easier than producing, so a model can be a useful critic even where it's a shaky author. The failure comes from forgetting that the judge is itself a model with a prompt, subject to biases you can name in advance.

Treat the judge as a system under test. Write a concrete rubric, prefer pairwise comparisons and low-cardinality scales, defend against position and verbosity and self-preference bias by construction, and — above all — calibrate against humans on a labeled sample before you believe a single number it produces. Do that, and you get evaluation that scales. Skip it, and you get a confident, consistent, and completely unaccountable machine for fooling yourself.