Researchers have developed OnTrack, a system designed to monitor the real-time performance of Large Language Model (LLM) agents. The system addresses issues of cost and safety that arise when autonomous agents take irreversible actions without adequate oversight. Current methods for monitoring LLM agents either involve a secondary agent that adds latency to every step or post-hoc log analysis, which is too late to prevent errors.
OnTrack operates by comparing an agent's current actions and their dependencies against a record of successful past executions. This comparison is performed as the agent progresses through its task. If a deviation is detected, OnTrack can alert a user or block the agent's operation. The system processes each step in approximately one millisecond.
The researchers studied OnTrack's effectiveness under three different access levels for reference data. The first regime, full reference access, includes historical runs of the task and information about the tools the agent can use. The second, intermediate regime, only provides access to tool schemas. The third regime, with no prior knowledge, relies solely on the steps being generated by the agent in real-time. As the amount of available reference data decreases, the system's ability to detect issues also reduces, ranging from identifying plan violations to detecting loops or repeated tool calls without progress.
The system models an agent's task progression as a directed acyclic graph (DAG), where each step is a node and dependencies between steps are edges. This graph is then compared to reference solution graphs using optimal transport. The researchers evaluated OnTrack using trajectories from SWE-bench, a benchmark for evaluating LLM agents on software engineering tasks. In tests focusing on the first eight steps of an agent's execution, OnTrack ranked failing trajectories below successful ones with an AUROC score of +0.057, outperforming content similarity-based approaches.
Furthermore, when an abort policy was implemented on top of OnTrack, it saved approximately 18% of the computational resources that would have been spent on agents heading toward failure. The system correctly identified failing runs in 83% of cases where an intervention was made. This real-time monitoring capability is intended to mitigate risks associated with autonomous LLM agents in applications such as trip planning, stock trading, and IT incident triage.