A new research suite called DiabAgent aims to advance the quantitative evaluation of medical language models in achieving sustained physiological control. The suite, detailed in a paper on arXiv, focuses on Type 1 diabetes as a specific application, where patients rely on exogenous insulin adjusted based on factors like meals, exercise, and glucose measurements. DiabAgent comprises three components: the Agentic Physiological Control (APC) framework, the DiabetSim simulator, and the SweetBench benchmark.
Type 1 diabetes is an autoimmune disorder where the pancreas ceases insulin production, necessitating external insulin to maintain blood glucose levels within a healthy range of 70–140 mg/dL. Uncontrolled glucose can lead to severe complications, including hypoglycemia (low blood glucose) which can cause dizziness, unconsciousness, or death, and hyperglycemia (high blood glucose) which can damage nerves, eyes, and lead to kidney and cardiovascular disease over time. Current "artificial pancreas" systems, also known as closed-loop automated insulin delivery systems, typically involve a continuous glucose monitor (CGM), a control algorithm, and an insulin pump. These systems aim to mimic the pancreas's function by adjusting insulin delivery based on real-time glucose readings.
The DiabAgent suite addresses a gap in current medical language model evaluation, which has primarily focused on clinical question answering rather than the sustained impact of interventions. The APC framework within DiabAgent is designed to control glucose through medical device tools. It uses ongoing observations to infer individual patient states, accumulate personal experience, and select appropriate actions. This approach is crucial for managing conditions like Type 1 diabetes, where continuous adjustments are necessary.
The DiabetSim simulator provides a testing environment for these AI agents. Simulators, such as the UVA/Padova metabolic simulator, are widely used in diabetes research and have been accepted by regulatory bodies like the U.S. Food and Drug Administration (FDA) for testing control strategies in lieu of animal trials. These simulators model the complex glucose-insulin regulatory system, allowing researchers to evaluate controller performance under various conditions, including different meal loads and unannounced meals.
The SweetBench benchmark offers a standardized method for quantitatively evaluating the performance of AI agents in this context. Benchmarks are essential for comparing different models and identifying areas for improvement, similar to how LiveAgentBench provides scenarios for general AI agents in real-world tasks.
Previous research has explored various control strategies for artificial pancreas systems, including multiple-model strategies and model predictive control (MPC), with a focus on improving time in range (TIR) and reducing hypoglycemic events. Efforts have also been made to develop "one-size-fits-all" artificial pancreas algorithms that can estimate insulin requirements based on body weight, CGM measurements, and meal carbohydrate estimates, without requiring individual customization by healthcare professionals. The development of AI-driven solutions for diabetes management, such as the BeaGL system at UC Davis Health, also points to the potential for AI to transform daily diabetes care by automating insulin dosing and reducing the patient's burden.
The DiabAgent suite aims to provide a structured approach for evaluating how well AI agents can manage the dynamic and personalized nature of physiological control. This includes their ability to reason over time, maintain persistent physiological context, and proactively monitor long-term signal streams, similar to the goals of other agentic frameworks like VitalAgent for wearable health data. The suite’s focus on sustained intervention and personalized experience reflects a move toward more autonomous and adaptive AI systems in healthcare.