A new study published on arXiv explores the limits of what can be known about the internal workings of complex systems, including artificial intelligence, through experimental observation. The research, titled "Same Behavior, Different Mechanism: What Finite Experiments Can Certify About Integrated Information," demonstrates that identical external behavior can arise from distinct internal computational processes. This distinction is critical for understanding AI systems, particularly in the fields of interpretability and auditing.
The paper focuses on integrated information, a measure known as $\varphi$ (phi), derived from Integrated Information Theory (IIT). IIT proposes that consciousness is linked to a system's intrinsic cause-effect power, quantified by $\varphi$. While IIT is often discussed in the context of consciousness, this study utilizes its mechanism-level quantity to investigate how much can be certified about a system's internal structure from experimental data. The researchers found that even when two systems exhibit identical input-output behavior, their internal mechanisms can differ substantially, leading to different $\varphi$ values. For instance, a recurrent network and a feed-forward network were shown to have the same input-output laws, yet one computed 0.28 bits of integrated information while the other computed 0 bits. These differences, however, were discernible with internal recordings within a limited number of trials (4 to 53).
The study highlights several challenges in inferring internal mechanisms from experiments. Partial access to a system's internal states can impose precision limits that cannot be overcome by increasing the sample size. In some cases, recordings of hidden units only reveal information through the product of their fidelities, a symmetry that can obscure differences without specific interventions. Furthermore, the measure of integrated information ($\varphi$) has "sharp edges," meaning small changes in the underlying mechanism can lead to significant changes in the calculated value. This sensitivity means that even models that closely fit interventional data might misreport the true $\varphi$ value.
Despite these limitations, the research suggests that finite experiments can still yield valuable insights. The authors proved that one-sided claims about $\varphi$ can be certified, meaning lower bounds can be established. For example, they demonstrated that a claim of $\varphi > 1.004$ bits could be certified from approximately $10^7$ controlled transitions with a low probability of error. This suggests that while sweeping claims about internal mechanisms may be unsupportable, more constrained assertions are certifiable. The work implies that for AI auditing, relying solely on input-output behavior is insufficient. Instead, researchers need to employ designed interventions, seek internal access where possible, and formulate claims that are narrow enough to be reliably supported by finite data. The study frames measurability not as an inherent property, but as a design question with computable answers, guiding how experiments should be structured to yield meaningful mechanistic insights.