Large language model (LLM) based medical assistants frequently encounter difficulties in maintaining a patient's accurate clinical state across multiple interactions, such as managing chronic conditions or providing medication guidance. A new research paper details MedGraph, a graph memory system that aims to improve the reliability of these AI assistants by explicitly addressing three common failure modes in long-term medical dialogue.
The MedGraph research team identified these recurring issues by examining 150 incorrect responses from five different memory systems on longitudinal medical dialogue. The errors typically arose from three sources: the retrieval of outdated patient information, the inclusion of health facts belonging to another individual, or the omission of critical safety facts relevant to the assistant's advice. For example, an LLM might recommend a discontinued medication if its memory provides an old record, or it might miss an allergy if that information is not prioritized or correctly linked to the patient.
MedGraph's design directly counters these problems. The system constructs a multi-layer memory by extracting dated clinical facts from each patient session. Each fact is attributed to the specific person it describes, and records related to medications, diagnoses, and measurements are linked into chronological chains. This structure allows the system to preserve earlier states while also tracking changes over time.
During retrieval, MedGraph ranks these patient-specific chains as units, reading each chain at the specific time the question refers to. It also incorporates the original dialogue text to retain nuance that might be lost in extracted facts. A patient profile, which lists every recorded safety fact, accompanies every question posed to the LLM, ensuring that critical safety information is consistently available.
The researchers evaluated MedGraph on MedMemoryBench, a benchmark for longitudinal medical dialogue. MedGraph achieved the highest overall accuracy with both GPT and Qwen backbones, two different large language models. The system surpassed the strongest baseline by 10.5 percentage points with the GPT backbone and 10.4 percentage points with the Qwen backbone. The paper also notes that MedGraph demonstrates efficient memory construction and answering within the GPT setting. This research suggests that focusing on structured state management for dynamic and critical medical facts is more effective than simply expanding the context window of an LLM.