Researchers have developed a new method to assess how consistently tool-using AI agents execute tasks across different languages, focusing on the sequence of actions rather than just the final outcome. The study, detailed in a paper titled "Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents" on arXiv, examines the "action policy" of eight models across six benchmarks and 41 languages, involving 2.38 million rollouts. This approach addresses a gap in traditional multilingual evaluations, which typically compare only final answers, disregarding the intermediate steps an agent takes.
The actions an AI agent performs determine factors such as cost, latency, and how the system might fail, making them a critical, auditable component of its behavior. The authors of the paper argue that focusing solely on final answers provides an incomplete picture of an agent's cross-lingual capabilities.
The research identifies five confounds that can distort naive measurements of trace similarity: shorter traces often score higher, empty traces achieve perfect scores, unrelated traces can agree by chance more than half the time, reproducibility limits the achievable gap, and models exhibit inconsistent behavior when asked the same question twice in one language. The researchers developed methods to remove these confounds, which, according to their findings, consistently amplified the observed effects.
The study found that divergence in agent actions across languages is a structural issue, not merely sampling noise. This divergence persists even with greedy decoding and remains stable as temperature settings increase, even as models become less self-consistent. When normalized by their own reproducibility, four distinct frontier models, despite their differences, converged under greedy decoding, retaining 71-73% of their action policy across languages. Model identity accounted for only 5.7% of the variance in these cases.
The consistency of action policies appears to break down in models smaller than approximately 10 billion parameters. For these smaller models, the observed ordering is largely an artifact of a chance floor, which the researchers measured through permutation rather than assumption.
A significant finding is that AI agents frequently route non-English tasks through English. This "pivot" to English is causally load-bearing, a conclusion confirmed by a pre-registered prediction across four models. The models did not abandon this English pivot even when instructed to do so. This suggests an underlying reliance on English as an intermediary language for processing multilingual requests.
The research also highlighted an instance where a single trace-extraction regular expression, rather than the model itself, caused a multilingual failure. Providing two worked examples increased one model's measured accuracy twenty-sixfold, while its accuracy on readable outputs showed minimal change. This indicates that external factors in how agent behavior is interpreted or extracted can significantly impact reported performance.
This work contributes to a broader understanding of how AI agents operate in multilingual environments. Previous research has explored how linguistic framing can influence cooperation in LLM agents in strategic scenarios like the Prisoner's Dilemma. Other studies have investigated implicit biases in language models by analyzing agent decisions. The ability of AI agents to maintain consistent policies across languages is increasingly relevant for global enterprises seeking to provide uniform customer support and operational predictability. Companies are developing multilingual AI agents that can detect language, switch between languages mid-conversation, and maintain context, aiming to ensure consistent policy application across diverse markets.
The current research underscores the need for more nuanced evaluation methods that go beyond surface-level outcomes to scrutinize the underlying actions of AI agents, particularly as these systems become more integrated into complex, real-world applications.