Your LLM Judge Is Not Neutral. It Has Opinions You Never Asked For.

Reading time: about 9 minutes.

A team ships a prompt change. The eval score goes from 0.82 to 0.86. Everyone relaxes, the change ships to everyone, and the meeting moves on. Nobody asks what actually produced that number.

Here's what usually produced it: a judge model, told to "rate this response from 1 to 10," scoring forty examples nobody has looked at in months. That number isn't evidence. It's a number that feels like evidence, and that's worse than having no number at all, because it ends the conversation before anyone asks a real question.

TL;DR

An LLM judge is an instrument, and an uncalibrated instrument gives you confident readings that are wrong in the same direction every time. Check the judge against real human labels before you let it decide anything.

• Judge bias is measured, not a guess. Researchers have catalogued twelve distinct biases in LLM judges, including favouring longer answers and favouring confident-sounding wrong reasoning. These come from the method itself, not from a bad prompt.
• Longer answers score higher, correct or not. Judge scores track answer length far more than human scores do. A prompt change that makes answers longer can look like a quality win when nothing actually improved.
• "Anonymise the outputs" doesn't fix self-preference bias. The real cause is that judges rate familiar-sounding text higher, not that they recognise their own writing. The fix is using a judge from a different model family.
• Offline and online evals answer different questions. One catches known problems before you ship. The other catches the ones you never thought to test for. Skip either one and you have a predictable blind spot.
• A test set left alone goes stale. Your traffic keeps changing while the test set sits still. Feed new failures back into it on a schedule, not whenever someone happens to remember.

The judge is supposed to be a neutral referee. It isn't. It has measurable preferences, and every decision built on top of it inherits them.

What This Article Answers

• Why does a green eval dashboard not mean your agent is actually good?
• What exactly is wrong with asking a judge to "rate this from 1 to 10"?
• Why doesn't anonymising the model fix judges favouring their own style?
• What's the real difference between offline and online evaluation?
• Why does a good-looking test suite quietly stop meaning anything over time?
• How do you actually check whether your judge can be trusted?

Two Kinds of Testing, Doing Two Different Jobs

Teams talk about "evaluation" like it's one thing. It's two, and mixing them up is why so many teams have a green dashboard and no real idea whether their agent works.

The first kind runs before anything ships, against a set of test cases with known-correct answers. Its job is catching regressions: did this change break something that used to work? It's fast, it can block a bad change from merging, and it's blind by design, because it can only catch problems someone already thought to test for. In practice this looks like a few hundred version-controlled test cases running automatically on every change.

The second kind runs against real, live traffic. Its job is finding what you didn't think to test for. A judge samples a slice of real production conversations and scores them against the same standard, and that ongoing signal is where you actually discover new failure modes, slow drift, and problems nobody predicted.

Run only one and you get a predictable failure. Test set only: your dashboard stays green while real users are having a bad time, because the problems moved and your test set didn't. Live traffic only: you find out about problems from the people they happened to.

The connection between the two is the part teams skip. Every new problem found in production should turn into a new test case. That's the only thing that keeps a test set from becoming a museum of last year's bugs.

Ask a Judge Model to Grade, and It Will Have Opinions

Using an AI model to grade another AI model's output is what made testing at scale affordable. It's also usually deployed with zero checking, and the biases are well documented enough now that skipping the check is a choice, not an oversight.

One well-known framework for measuring this, CALM, catalogued twelve separate biases: things like favouring earlier answers in a comparison, favouring longer answers, favouring confident phrasing, and giving credit for a correct final answer even when the reasoning behind it was wrong. A few of these matter a lot for testing an agent specifically.

Length. Judges reward longer answers. In measurements, judge scores track raw answer length far more closely than human scores do. If a prompt change made your agent's answers longer, your judge score can go up even though nothing actually got better. Teams optimise against a judge for months and end up with an agent that just talks more, not one that's more correct.

Accepting broken reasoning. A judge will often give full credit for a correct final answer, even when the steps that produced it were wrong. This was one of the weakest spots measured across the models tested. For an agent this matters a lot, because the reasoning is the actual product. A judge that only checks the final answer will pass an agent that got lucky, and getting lucky once doesn't mean it'll happen again.

Fake citations. A judge will often treat the mere presence of a citation as a quality signal, even a fabricated one. An agent that invents a plausible-sounding source gets rewarded for it.

Order. In side-by-side comparisons, which answer comes first changes the result. The fix is simple and non-negotiable: run the comparison both ways and throw out or flag anything where the answer changed.

One honest caveat: the specific numbers behind these findings come from older, 2024-era models, so treat the pattern as the real finding and the exact figures as a snapshot in time. Newer research that goes looking for biases without assuming what it'll find suggests twelve is probably a floor, not the whole list.

"Just Anonymise It" Doesn't Actually Fix Anything

There's a common explanation for why judges favour certain answers, and it leads people to a fix that doesn't work.

The usual story: a judge model favours its own outputs, so hide which model wrote what before grading. Reasonable idea. It's also wrong about the mechanism. Research on self-preference bias found that judges give higher scores to text that's easier to predict, whether or not the judge itself generated that text. The real driver is familiarity, not authorship. A model just finds its own style of writing easy to predict, so easy-to-predict text from anywhere scores well.

That changes what you should actually do. Hiding the source fixes nothing, because the source was never the real cause.

What actually works: use a judge from a different model family than the system you're testing, so "this sounds like something I'd write" and "this is actually correct" stop being the same thing. Treat fluent, confident-sounding text as a warning sign to check more carefully, not a reason to trust it. And if your eval scores shift right after you swap models, be suspicious. You may be measuring how familiar the new judge finds the text, not whether it actually got better.

# A judge that just returns a number isn't useful.
# Make it show its work, then check the judge itself.

verdict = judge(
    rubric=explicit_criteria,        # never "rate this 1 to 10"
    require=["failed_criteria", "evidence_span", "pass_or_fail"],
)

# order bias: grade both orderings, keep only the ones that agree
if judge(a, b).winner != flip(judge(b, a).winner):
    return DISAGREEMENT            # send to a human, don't just average it away

# the one number that tells you if any of this can be trusted
agreement = cohens_kappa(judge_labels, human_labels_on_same_sample)

That last line is the whole point. A judge you've never checked against real human judgement on your own task isn't a testing instrument. It's a guess with an API attached. And trustworthiness isn't really about which model you pick, it depends heavily on the specific task, so "this model is a good judge" doesn't automatically carry over from one use case to another. Take a few hundred real cases, label them by hand, measure how often the judge agrees with the human, and check again whenever you change the judge model or your traffic shifts.

Give the Judge a Question It Can Actually Answer

Most judge prompts fail before any bias even shows up, because they ask a question with no real answer.

"Rate this from 1 to 10" asks a model to squash correctness, helpfulness, tone, and format into one number with no defined scale. Nobody can say what separates a 7 from an 8, including the judge. It answers anyway, because it always answers, and that answer ends up driven by surface features: length, smoothness, how confident it sounds.

Break the question apart instead. Turn quality into a handful of specific, checkable criteria: did it use the retrieved information, did it make anything up, did it follow the required format, did it refuse when it should have. Pass or fail on each one. Make the judge point to the specific part of the answer that supports its verdict; this makes the grading both better and checkable afterward. Save the judge for the parts only a language model can actually assess.

That last point is worth dwelling on. A lot of what teams hand to a judge could just be checked directly in code: does the format match the schema, are required fields present, does it avoid banned content, do cited sources actually exist, are numbers in a sane range. Direct checks like these are free, fast, and can't develop a length preference.

Then there's the test set itself, which is where the real, boring failure usually lives. A test set is frozen the moment it's written, while your inputs, users, and other systems keep changing around it. Coverage quietly goes stale, the dashboard stays green, and green stops meaning what it used to. The fixes aren't exciting: sample real production traffic on a schedule and add new cases, watch for live inputs drifting away from what your test set covers, and treat every real incident as a mandatory new test case. That last habit is the cheapest thing in this whole series to do and the one most often skipped.

This is the same shape of problem covered in Part 6, where a single end-to-end score hides which specific stage actually failed. Here, one overall eval score hides which type of request is actually failing, and the overall number is exactly the one that gets quoted back in the meeting.

A Number Ends the Argument, Which Is the Problem

There's a people problem here worth naming directly, because it's the real reason bad tests survive so long.

A number ends debate. Before a dashboard exists, shipping a change means someone has to argue it's actually better, and that invites pushback. Once the dashboard exists, the score moved from 0.82 to 0.86 and the conversation is over. The test has become something people defer to, rather than a tool people question.

This is exactly how a forty-example spreadsheet graded by an unchecked judge ends up controlling a live product for two years straight. Not because anyone thought it was rigorous. Because nobody ever asked it to prove itself after the first week. And the same score quietly sits underneath both the routing decisions and the prompt promotion decisions covered earlier in this series, so every bias hiding inside it flows straight into both.

The fix is making the test answer to something. Report how well it agrees with real human judgement right next to the score, so people see the uncertainty along with the number. Recheck it on a schedule. And keep a handful of cases where you know the right answer and know the judge gets it wrong, as a standing reminder of what it actually can't see.

The Test Itself Is the One Thing Nobody Tests

Every other part of a production system in this series gets checked by an eval. The eval itself gets checked by nobody, which makes it the least examined and most blindly trusted piece of the whole thing.

An LLM judge is genuinely useful and genuinely not an oracle. It has real, documented biases, all pulling in the same direction: toward length, toward smooth confident phrasing, toward answers that sound like something the judge itself would say. None of that makes it useless. It just means it's something you have to check, not something you get to assume.

If you can't say how well your judge actually agrees with a real person on your own task, you don't have a testing system. You have a random number generator with good manners.

Coming Up in This Series

Next up: Security: prompt injection, the confused deputy, and data leaking out the side door. That wraps up the quality block and opens the trust block. Every capability covered so far is also a new way to get attacked: tools that take real actions, retrieval that can pull in text an attacker planted, and an agent that often can't reliably tell an instruction apart from data it's just reading. Most systems right now have one validation function standing between them and a bad day.

---

If this resonated and you're building production AI systems, follow along. The series covers the 21 things I think senior AI engineers and architects need to reason about: RAG pipelines, tool design, security, evaluation, cost, and the operational patterns that separate demos from systems you can actually run.