A new benchmark, VAKRA (eValuating API and Knowledge Retrieval Agents), has been introduced to assess the multi-hop reasoning capabilities of artificial intelligence agents in complex enterprise settings. Developed by researchers, VAKRA evaluates how AI systems can navigate and utilize over 8,000 executable APIs across 62 different domains, alongside document collections, to complete multi-step tasks. This benchmark aims to bridge the gap between isolated skill evaluations and the demands of real-world applications where agents must chain decisions, reconcile data from various sources, and adhere to natural-language tool-use policies.
Existing benchmarks often test AI capabilities in isolation, such as single API calls or simple question answering. VAKRA, however, focuses on compositional reasoning, requiring agents to execute multi-step workflows that mimic enterprise scenarios like customer support, business intelligence, and compliance. The benchmark includes tasks that involve reasoning chains of up to seven steps, combining structured API interactions with unstructured document retrieval. Correctness is verified through the re-execution of predicted tool calls against live APIs, allowing for multiple valid paths to a solution, which is crucial for enterprise workflows.
Researchers have found that even advanced AI models struggle with the complexity introduced by VAKRA. When using a fixed ReAct harness to isolate model capabilities from agent architecture, the best-performing models achieved approximately 70.4% accuracy on single-hop tasks. However, this performance dropped to around 50-51% on compositional API tasks. The study also revealed that performance degrades by over 50% as the depth of reasoning increases. Furthermore, tasks involving natural-language tool-use policy constraints exposed severe failures, with accuracy as low as 2.4% on unanswerable queries. Trace analysis indicates that failures often occur in the language-mediated reasoning steps, such as entity disambiguation and cross-source grounding, rather than in the mechanics of tool invocation.
The VAKRA benchmark is designed to provide a standardized method for evaluating AI agents' suitability for multi-hop API and retrieval tasks under policy constraints. Its creators emphasize that enterprise environments rarely involve simple, single-turn interactions. Instead, agents must effectively chain decisions, align disparate data schemas, and interpret constraints expressed in natural language. For example, resolving a delayed order complaint in an e-commerce operation might require an agent to link customer records, interpret carrier documentation, and reconcile identifiers across logistics APIs, all while adhering to specific policies.
The benchmark is made available as open-source code, allowing researchers and developers to run agents, reproduce results, and evaluate new systems end-to-end. This initiative aims to drive progress in developing more reliable and capable AI agents for complex, real-world applications. The low performance observed on policy-constrained tasks, particularly for unanswerable queries, highlights a potential risk in production systems where agents might hallucinate or take incorrect actions when faced with such restrictions. Future work will likely focus on improving agent architectures and model capabilities to better handle these intricate reasoning chains and policy adherence requirements.