Researchers have introduced EpiSEM-Bench, a new benchmark designed to evaluate the safety and evidence recovery capabilities of large language model (LLM) agents that utilize tools. The benchmark addresses a critical scenario where tool responses may omit safety-relevant facts, potentially leading an agent to execute a prohibited action even when the user's request is benign and all provided statements are true. EpiSEM-Bench consists of 300 matched unsafe and safe task pairs distributed across six distinct domains.
The evaluation method involves systematically suppressing key facts within tool responses. This approach allows the benchmark to measure unsafe execution under conditions of critical evidence omission. EpiSEM-Bench also determines the minimum observed failure cardinality, which includes instances of failure even when full evidence is available, as well as cases where no failure is observed after exhaustive testing. An independent audit tool is integrated to assess whether agents actively seek out missing facts and subsequently refrain from prohibited actions while still completing permissible tasks.
Testing across ten different LLM models revealed significant variations in their robustness to evidence omission, their ability to acquire missing information, and their capacity for appropriate action. The research indicates that an agent's resistance to missing evidence does not automatically translate into effective recovery of that evidence. While providing access to recovery mechanisms reduced unsafe execution for all ten models tested, the agents differed in their proactive acquisition of missing facts and their subsequent appropriate behavior in both prohibited and permissible scenarios. For example, one study noted that DeepSeek-V4.1-Flash had a low premature-commit rate of 0.8% with 86.7% strict unsafe recovery, while Qwen3-14B exhibited a high premature-commit rate of 72.0% in unsafe recoverable evidence worlds.
The introduction of EpiSEM-Bench aims to provide a standardized method for quantifying and improving the safety boundaries of LLM agents. It seeks to ensure that these agents do not perform harmful actions due to incomplete information, while simultaneously maintaining their utility in completing valid tasks. The benchmark is intended for teams developing tool-using agents, safety evaluators, and anyone concerned with the reliability of AI systems operating with potentially incomplete data.