AI evaluation has become increasingly sophisticated at testing individual models. We measure accuracy, test hallucinations, run fairness checks, evaluate robustness, and compare systems against increasingly demanding benchmarks.
Yet when we studied real-world AI failures, one pattern kept appearing: many of the failures that mattered most were not really model failures alone. They emerged after the model entered a real operational environment.
A model can perform well during testing and still create problems because the data is incomplete, the workflow is poorly designed, monitoring is weak, or nobody can realistically intervene once something goes wrong.
To understand whether this pattern was random or systematic, we examined 1,368 documented real-world AI incidents and asked whether these failures had recurring features.
Failures rarely stay inside the model
A technical problem might begin with poor data, weak predictive performance, or an overlooked failure mode, but whether that weakness became harmful often depended on what happened after deployment.
The consequences depended on AI authority, meaningful human intervention, business usage, and whether anyone noticed the problem early enough.
That distinction also appeared in our analysis. Severe monitoring-and-detection failures were associated with a 4.7-times lift in fatal outcomes, while high AI decision authority was associated with a 2.8-times lift. These are associations within reported incidents, not causal estimates, and the underlying data is subject to reporting bias. Even so, they raise an important question: what if consequential AI risks are shaped not only by what a model gets wrong, but by what the surrounding system allows to happen afterward?
What this looks like in practice
Consider Michigan's MiDAS unemployment system, which automatically issued fraud findings and penalties. Michigan's unemployment agency later reviewed 40,195 fraud findings resolved by computer program alone and reversed 85 percent of them. It also reviewed 22,589 cases initiated by computer and then referred to an investigator, reversing 44 percent of those findings.
The lesson is not simply that automation made mistakes; human involvement alone did not make the problem disappear either. The better question is whether the process provides a meaningful opportunity to detect and correct errors before consequences reach people.
A similar pattern appeared in a South Australian aged-care monitoring pilot. An independent review found that the system generated more than 12,000 alerts that were not verified as true events. During the final months, staff could no longer respond to every alert, and at least one unanswered alert involved an actual resident fall.
A related question arises in an ongoing U.S. class action involving UnitedHealth. Plaintiffs allege that staff were expected to remain within one percent of the care duration predicted by an algorithm. The defendants dispute the allegations, the litigation remains ongoing, and no liability has been established.
Regardless of how that case is resolved, it illustrates a broader question: when an AI system is wrong, can a human realistically disagree with it and change the outcome?
Evaluation needs a second question
Traditional evaluation still needs to ask whether the model works under expected conditions. That question remains essential, but real-world incidents suggest it cannot be the only question we ask.
We also need to ask: under what conditions can the system fail, and what happens when that failure reaches the real world? That question shifts evaluation toward actively searching for conditions where our confidence might break.
What happens when the underlying data changes over time? What happens when thousands of alerts arrive faster than humans can meaningfully review them? What happens when a human can technically override the AI, but organizational pressures make that override difficult in practice?
From model evals to system evals
Model evaluation is not becoming less important. It is becoming one layer of a larger evaluation problem.
We think about that problem through three connected questions. Where can the system break across data, design, predictive validity, fairness, explainability, and failure modes? Can that failure reach the real world through excessive authority, weak oversight, limited reversibility, or inadequate monitoring? How much does the failure matter in financial, legal, operational, safety, reputational, or human terms?
Not every technical weakness deserves equal attention. The goal should not simply be to identify more possible failures. The more important goal is to identify the failures that matter, understand how they can travel through a system, and determine whether enough controls exist to stop them before they become real harm.
That becomes increasingly important as AI systems move from recommending what people should do toward taking more consequential actions themselves.
The next generation of AI evaluation will require better benchmarks and stronger model-level measurement. It will also require something broader: the ability to evaluate the entire system around the model and discover how it can fail before the real world discovers those failures for us.