A new approach dubbed "cache engineering" offers Large Language Model (LLM) agents a way to manage their memory more efficiently. Developed by researchers at Tsinghua University and The Chinese University of Hong Kong, this method allows agents to directly control their active key-value (KV) cache. This control is achieved without altering the sequence of actions and observations, known as the trajectory, and crucially, without triggering costly re-computation of cached data.
As LLM agents handle more complex tasks, their trajectories grow longer. This expansion leads to increased costs associated with KV cache storage and input/output operations. Furthermore, trajectories can exceed the maximum context length of an LLM, potentially degrading task accuracy. Prior methods, referred to as "context engineering," attempted to mitigate these issues by trimming or summarizing past information. However, modifying the trajectory prefix invalidates subsequent cached states, necessitating a re-computation of the retained context. This process, known as re-prefilling, can be expensive, especially if significant portions of the history must be reprocessed.
The proposed cache engineering scheme bypasses this limitation. Instead of altering the textual history, agents can select specific KV entries to exclude from future attention computations. This direct manipulation of the cache allows for incremental dropping of older entries while retaining more intermediate context. The researchers have developed an efficient serving engine, named AKV, to support this cache engineering. AKV provides a simple interface compatible with existing API-based serving systems, managing the complexity of stateful cache operations internally.
Experiments detailed in the paper demonstrate that AKV enables a "rolling-drop" strategy for cache management. This strategy has shown improvements in benchmark scores by an average of 1.7 points, while simultaneously reducing average costs by 27.7% across various LLMs and agent benchmarks. The research highlights that KV cache reads can account for a significant portion of total API costs, reaching as high as 82.4% in some scenarios. By optimizing cache usage, the new method addresses a critical bottleneck in the deployment of advanced LLM agents.
The challenge lies in the history-dependent nature of cache states. The same sequence of tokens can result in different cache states depending on the history of cache operations performed. This makes simple token-prefix matching insufficient for determining cache reusability. AKV addresses this by extending radix tree structures and attention masking techniques to maximize cache reuse, enable timely eviction of data, and efficiently restore any necessary cache states. The researchers suggest that this capability could further enable recursive self-improvement in agents.