Researchers have identified critical privacy and security vulnerabilities in a publicly available medical artificial intelligence chatbot that uses retrieval-augmented generation (RAG) technology. The study, detailed in a paper on arXiv, found that the chatbot's internal operations and data handling mechanisms were exposed through its public interface, posing risks to sensitive patient information.
The investigation employed a two-stage approach. Initially, researchers used generative AI, specifically Claude Opus 4.6, to explore potential weaknesses and formulate hypotheses about vulnerabilities. This exploratory phase was followed by manual verification using browser developer tools, which allowed the researchers to inspect network traffic and data payloads transmitted by the chatbot. This method enabled them to observe how the chatbot interacted with its backend systems.
The core issue identified relates to the architecture of RAG chatbots in healthcare. These systems are designed to retrieve information from a knowledge base to generate responses, aiming to provide accurate and grounded health information. However, the study suggests that the implementation of these systems can inadvertently create pathways for data exposure. Even with efforts to avoid training models on protected health information (PHI) or other sensitive data, architectural flaws can lead to incorrect exposure through RAG processes.
The research highlights a broader concern within the healthcare technology sector regarding the security of AI applications. A report by ECRI in February 2026 identified the misuse of AI chatbots, including general-purpose ones like ChatGPT and Gemini, as the most significant health technology hazard for that year. While these general chatbots are not FDA-approved medical devices, their integration into healthcare workflows and use by individuals seeking health advice present considerable risks.
The paper on arXiv points out that RAG systems in healthcare can involve a wide array of sensitive data, from basic clinical information to more complex genomic and biomarker data, especially in precision medicine initiatives. The security and privacy challenges associated with handling such granular and sensitive data are substantial. A key risk involves the accidental introduction of PHI into AI models or the incorrect exposure of such information through retrieval mechanisms.
Furthermore, the study indicates that users often lack awareness regarding how their data is stored and shared when interacting with chatbots. In many cases, healthcare chatbots do not fall under HIPAA compliance, meaning user data could potentially be shared, sold, or stored without explicit user knowledge or consent, further exacerbating privacy risks. The continuous machine learning process of AI chatbots involves feeding data back into their neural networks, meaning sensitive patient or business information, if compromised, could become part of the chatbot's future interactions and data sets.
The researchers' findings underscore the need for rigorous security, privacy, and governance controls for AI systems deployed in healthcare. While AI-assisted development lowers the barrier to creating these tools, it does not negate the requirement for robust safeguards. The exposure of backend processes in the studied chatbot suggests a failure in these necessary controls, allowing for potential breaches of patient confidentiality.
The study did not involve destructive testing, implying that the vulnerabilities identified were accessed without causing harm to the chatbot's operational integrity. However, the implications for patient data security are significant. The research team aims to provide governance lessons for the safe deployment of generative AI in the health sector, emphasizing that as AI adoption grows, so too must the focus on protecting sensitive medical information. The next steps in this area will likely involve developing and implementing more stringent security protocols and oversight mechanisms for medical AI applications.