The MindEval2 framework, detailed in a paper published on arXiv, aims to improve the evaluation of AI chatbots in mental health support. The framework consists of two main components: Mental Health Quality Rubrics (MHQR) for evaluation and MindSim for patient simulation. These components were developed to overcome the limitations of existing benchmarks, which often rely on overly forthcoming simulated patients and opaque Likert scales for judgment.
The MHQR component was developed in collaboration with PhD-level psychologists. It establishes a set of rubrics that break down the quality of a language model's response into explicit, operational criteria. These criteria are designed to be more interpretable and actionable than previous evaluation taxonomies. The rubrics assess 14 clinical issues that a counselor's response might exhibit, with each issue graded by severity. This approach allows for a detailed assessment of individual counselor turns within a conversation.
MindSim, the second component, is an LLM harness that creates more varied and challenging patient interactions. It uses carefully designed patient profiles, including persona, clinical anchor, core belief, and program goals. Unlike earlier simulation methods, MindSim incorporates a memory component that updates between interactions, allowing MindEval2 to model the longitudinal aspect of mental health support. This feature enables the framework to test how language models respond to patient hesitation, partial disclosure, or resistance, and how they utilize context from prior sessions. Previous interactive benchmarks often focused on single sessions, overlooking the need to evaluate a model across multiple interactions with the same patient.
The researchers evaluated 18 proprietary and open-weight language models using MindEval2. The findings indicated that while advanced models have shown improvements in surface-level quality and boundary setting, "guidance calibration" remains a primary failure mode across all tested models. This failure mode accounts for most of the observed quality gap in their performance.
The development of MindEval2 builds upon earlier efforts to benchmark language models in mental health support, such as the initial MindEval framework. The earlier MindEval, introduced in November 2025, also used patient simulation and automatic evaluation with LLMs, and was designed in collaboration with clinical psychologists. However, MindEval2 specifically addresses the need for more realistic patient behaviors and clearer evaluation criteria, moving beyond the limitations of its predecessor. The code, prompts, benchmark profiles, and annotations for MindEval2 are publicly available on GitHub.