Large language models (LLMs) employed as "judges" for tasks such as filtering training data, scoring generated content, and populating leaderboards demonstrate unreliable and inconsistent behavior, according to a new study published on arXiv. The research, titled "Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints," found that the assumption of consistent output from these models is often not met.
The study, conducted by Haoyuan Zhu and Jie Zhang, involved two preregistered audits with predefined thresholds, analyzing 52,988 request attempts to black-box LLM observers. The researchers aimed to validate the stability of these LLM judges as measurement instruments. Their findings indicate that the models failed to meet the established reliability criteria.
Specifically, repeat rankings of identical requests made within the same short time frame showed a Spearman rank correlation of 0.400. This figure is considerably lower than the required 0.90 threshold set by the researchers for acceptable agreement. When the same byte-identical inputs were replayed the following day, the agreement reached 0.78, still below the stipulated 0.99 requirement.
The authors identified three primary mechanisms contributing to this instability. First, a "label-to-meaning mapping" introduced bias into the readouts, influencing the results as strongly as the actual signal being measured. Second, the differences between candidate outputs were often seven orders of magnitude smaller than the inherent noise floor of the LLM instrument itself. Finally, even byte-identical inputs sometimes produced different rankings, a form of noise that became more pronounced with exact-permutation readouts.
These findings have implications for the widespread practice of using LLMs in evaluation pipelines. Many model rankings, version comparisons, and benchmark tables in technical publications rely on automated evaluation running on shared endpoints. If the same request to the same model name yields different orderings in a significant percentage of re-executions, then small differences between competing models may be indistinguishable from measurement noise.
The research team explored several potential solutions to mitigate the observed instability. They found that waiting did not improve replay stability over the sampled days, with results of 0.805 versus 0.800 across five additional days. Switching among four different LLM providers also did not resolve the issue; the median correlations for these providers ranged from 0.74 to 0.88, none reaching the 0.99 replay gate. Self-hosting on batch-invariant kernels offered help only when the server load was low. Experiments with constructed errors revealed that the judge's separation tracked the type of error rather than the magnitude of the gap.
The authors conclude that on a shared endpoint, an LLM name does not represent a fixed instrument. They recommend that any preregistered evaluation should first assess the stability of its judging instrument before using it to set any thresholds. The paper distills its evidence into a three-level "snapshot-identity ladder," eight design rules, and a reporting checklist. A small pilot study, using roughly 2% of the call volume of their main audit, would have been sufficient to expose these reliability issues in advance.
Previous research has also highlighted concerns about the reliability of LLMs as judges. A 2026 RAND Corporation study indicated that no single LLM judge was uniformly reliable across all benchmarks. Other studies from 2025 and early 2026 have documented instances where LLM judges fail, including issues with prompt sensitivity, verbosity bias, and miscalibration. For instance, simple text formatting changes have been shown to disrupt consistency among judges that otherwise passed standard accuracy checks. The current study reinforces these concerns, providing empirical data on the inherent instability of LLM observers on shared infrastructure.