Benchmarking five large language models in medical genetics: a bilingual comparative evaluation using published and novel expert-authored questions.

This study asked whether five contemporary large language models answer medical genetics multiple-choice questions with equivalent accuracy on published versus novel items and across English and Turkish, and sought to characterize the errors that persist. Five models (GPT-5.2, Gemini 3 Pro, Claude Sonnet 4.6, Grok 4, and DeepSeek-V3.2) answered 100 four-option questions (50 from a published board review; 50 novel, expert-authored items absent from any database) in English and Turkish, yielding 1,000 responses. Correctness was modeled with item-clustered generalized estimating equation and Bayesian mixed-effects logistic regression (the latter as a prespecified sensitivity analysis); question provenance and language were tested for equivalence (item-clustered two one-sided tests, ±5-percentage-point margin), and the paired language effect with the McNemar test. Inter-model agreement and error concordance were examined. Overall accuracy was 97.7%. In the item-clustered GEE, only Gemini 3 Pro nominally exceeded the lowest-performing model; this imprecise contrast did not remain significant after Holm correction for the four secondary model comparisons, whereas the Bayesian sensitivity analysis additionally yielded a credible interval excluding 1 for GPT-5.2 versus Grok 4. Accuracy was statistically equivalent within the prespecified margin for published versus novel items (difference, -1.4 percentage points; item-clustered 90% CI, -4.1 to +1.3; item-clustered TOST