Researchers from the University of Deusto in Bilbao, Spain, assessed the performance of four open-source large language models (LLMs) on the Spanish Médico Interno Residente (MIR) examination, a high-stakes test for medical residency placement. The study, published on arXiv, examined models ranging from 8 to 27 billion parameters, scoring 1,552 questions from eight past editions of the exam. The three larger models demonstrated performance that would place them among the top 600 to 3,000 candidates, leaving 41 to 43 specialties open for them.
The MIR examination is a national medical test that determines eligibility for competitive medical specialist training positions in Spain. The study specifically investigated factors beyond simple accuracy, focusing on whether these open models could identify when they were likely to be incorrect, if medical fine-tuning improved this self-awareness, whether their correct answers stemmed from sound clinical reasoning, and if they effectively utilized provided images.
The researchers found that abstaining from answering questions, even those with low confidence, did not benefit the models, as their least confident answers were still correct more often than the 25% break-even point for random guessing. Medical fine-tuning, while intended to improve performance, paradoxically reduced overconfidence at the expense of overall accuracy. It did not enhance the models' ability to distinguish between correct and incorrect answers based on their confidence levels.
A significant finding was that between 13% and 38% of the models' correct answers relied on clinically unsound reasoning, a deficiency not reflected in their confidence scores. Furthermore, the study revealed that removing or altering the contextual images provided with certain questions did not affect the models' accuracy, suggesting a limited or absent use of visual information in their decision-making processes for these particular questions.
Previous research has also explored LLM performance on medical examinations. A 2024 study on the MIR 2024 and 2025 examinations highlighted significant performance variations among 22 LLMs, with a fine-tuned model, Miri Pro, achieving high accuracy, while general models struggled with complex clinical scenarios. Another evaluation of GPT-4o and OpenAI o1 on the MIR 2024 exam found both models achieved over 90% accuracy, outperforming the average human candidate. However, these studies often utilized larger, proprietary models. The current study specifically focused on open models that could be run on a hospital's own hardware, allowing for greater control and auditability.
The findings indicate that while open-source LLMs show promise in medical knowledge assessment, their current limitations in discerning uncertainty, consistently employing sound clinical reasoning, and effectively integrating multimodal information like images warrant further development before widespread clinical deployment.