ArticleBMC medical education2026
Consistency over accuracy: run-to-run stability of contemporary large language models on Turkish curriculum-aligned theoretical anatomy multiple-choice questions.
Article in BMC medical education, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 2 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
2 citing papers in PubMed.
- Performance of Reasoning and Comparator Large Language Models on Nephrology Multiple-choice Questions.JMA journal · 2026Article
- Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing.Journal of the American Medical Informatics Association : JAMIA · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
2 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundStability across repeated administrations is essential for educational use of large language models (LLMs), yet it is rarely quantified in non-English, curriculum-aligned anatomy contexts.
methodsEleven contemporary LLMs answered 100 Turkish, faculty-authored, curriculum-aligned anatomy multiple-choice questions from AYDEP, targeting the undergraduate Physiotherapy and Rehabilitation anatomy curriculum in three independent runs (≥ 12-hour intervals). Testing used developers' web interfaces in Turkey with browsing disabled and default generation settings (August-September 2025). Performance was summarized with a stability-aware 0-3 item score (number of correct responses across three runs) and predefined response-consistency classes.
resultsA subset of models achieved near-ceiling totals with high run-to-run stability, whereas others showed greater session-to-session variability (i.e., changes in the selected option across independent runs initiated as separate sessions under identical inputs). Several nominal differences among higher-performing systems did not remain significant after multiplicity control. Within-family updates produced selective, not universal, gains. Many models exhibited medians of 3 (IQR 3-3) on the 0-3 scale (ceiling effects), and lower means were accompanied by larger dispersion. Consistency profiles provided information beyond mean accuracy by distinguishing reliably correct from volatile behavior. In addition, we observed "consistent & wrong" patterns on a subset of items, where the same incorrect option was repeatedly selected across runs.
conclusionIn Turkish, curriculum-aligned anatomy items, contemporary LLMs can be both accurate and stable, but single-trial accuracy can mask volatility and stable systematic errors. Adoption decisions should prioritize stability-aware appraisal (including consistent-correct and consistent-wrong rates), with local validation on institutional item banks and periodic re-evaluation as models evolve. Extending this framework to multimodal anatomy and constructed-response tasks will further inform trustworthy, learner-facing use.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.