We’re increasingly asking AI to help decide whether other AI systems are good enough to ship. LLM judges help us improve prompts and choose models, and don’t get me wrong, they’re useful tools. But I keep coming back to one question: when we revise an eval, are we discovering a better definition of quality, or getting used to what the model happens to produce? Our feedback shapes the model, while its outputs can shape what we’re willing to accept. Understanding how that happens means looking at the human side of evaluation, and checking whether a rising score reflects better work or a changing standard.
Suppose we have a coding agent asked to fix a small bug. The solution works and the tests pass, but the agent has also rewritten unrelated parts of the module. A judge assessing correctness and readability could approve the result, while the engineer reviewing it faces extra work and possible conflicts with another branch. We might include that review cost in the evaluation, or engineers might gradually stop counting it against the agent because they’ve seen so many similar patches that they seem normal.
Does that really happen?
In Who Validates the Validators?, Shankar and colleagues studied practitioners developing evaluations. As participants reviewed outputs, they added requirements and reinterpreted existing ones, a process the authors call “criteria drift”. For example, two participants relaxed a requirement that every extracted entity be a proper noun, accepting outputs where most were. Now, their original rule may have been too strict; the point isn’t that the revision made things better or worse in this case. Instead, it does show that the human criteria we use to calibrate a judge can change in response to the behaviour they’re supposed to assess.
Another study, Just Put a Human in the Loop?, shows what this can do to measured performance. Researchers asked people to label themes in community discussions, with or without LLM suggestions. Against independent human labels, GPT-4 achieved an average weighted F1 score of 0.47. Against labels produced by people shown its suggestions, that rose to 0.79. GPT-4 had assigned exactly the same labels in both comparisons; its score rose when the human labels used to assess it came from people who had seen its suggestions. Each reference label required a three-out-of-five human majority, so multiple reviewers didn’t guarantee independence either. This shows that a model can appear to improve because it has influenced the humans providing the reference.
That influence can extend beyond accepting the suggestion currently on screen. In experiments by Glickman and Sharot, repeated exposure to a biased algorithm changed participants’ own initial estimates on later trials, before they saw the next AI answer. Accurate AI advice also improved human judgements. These were perception experiments, not engineers maintaining evals, but they give us a reason to examine how reviewers learn from the systems they assess.
What can we do about it?
I think the practical response is to separate developing an eval (LLM-as-a-Judge or otherwise) from checking whether it works. Looking at outputs is useful when we’re deciding what matters, but we should reserve fresh examples and some reviewers who weren’t involved in tuning the rubric. On that review sample, have people record their judgement before revealing the judge’s score or explanation. Keep both the initial judgement and any revision, rather than saving only the final consensus. That won’t erase previous exposure to AI, but it lets us see how this particular recommendation changes a decision. Agreement reached after reading the judge’s reasoning should be distinguishable from agreement reached independently.
We should also make changes to the standard visible. This is already pretty commonplace, but before adopting a revised rubric, run both versions over the same saved outputs and inspect the cases that switch from fail to pass, or vice versa. When comparing application versions, use the same tasks and hold the evaluator fixed. For example, if a score rises because we now permit broader rewrites, record why those rewrites are acceptable and what evidence supports that decision. Perhaps they reduce later maintenance work; perhaps the original scope rule rejected useful fixes.
The Product Hypothesis
There’s still the question of whether the feature is worth using. A well-calibrated judge can faithfully enforce an incomplete definition of quality. Our coding agent might now produce focused, readable patches while developers spend so long preparing context and checking the result that they could have finished sooner themselves. If we’re claiming to save time, we should measure comparable tasks with and without the agent, including setup, review, rework and unsuccessful attempts.
Netflix made a similar comparison in its work on AI-generated recommendation explanations. LLM judges checked whether the explanations met the team’s quality criteria, while a separate A/B test compared viewers who saw approved explanations with viewers who saw none. The test found improvements in content discovery and in browsing sessions that led to watching. Crucially, it could have found no benefit even if every explanation passed the judges’ checks. That gave the team a way to test the assumption underlying its evaluation: that producing explanations it considered good would actually help people choose something to watch. There was no space for the model outputs to distort our perception.
As more development gets automated, a judgement made by a handful of reviewers can become a rule applied to thousands of outputs, then a target for the next round of prompt tuning. A compromise made during review can end up shaping subsequent versions of the feature. That’s why I’d want to preserve the evidence behind a change in standards, including independent assessments that challenged it. When scores rise but the promised benefits fail to appear, we need to be able to revisit the decision that made those outputs count as progress.