Large language models for late-life depression: a blinded benchmark of clinical safety, geriatric appropriateness, and triage.

Background: General-purpose large language models are increasingly used by patients and caregivers to obtain mental health information and guidance about when professional care is required. In late-life depression, broadly accurate information may nevertheless be unsafe when cognitive change, multimorbidity, frailty, polypharmacy, self-neglect, caregiver dependence, or suicide risk is not adequately recognized. This study compared the clinical accuracy, safety, geriatric-specific appropriateness, triage performance, and response consistency of three large language models when answering patient- and caregiver-centered questions about late-life depression. Methods: We conducted a blinded, paired benchmarking study using 90 questions covering six geriatric psychiatry domains and equally distributed across low-, moderate-, and high-risk strata. Each question was submitted independently to GPT-5.5 Instant via ChatGPT, Gemini 3.5 Flash via Gemini, and Seed2.0 Pro via Doubao, generating 270 primary-round responses. A stratified subset of 30 questions was resubmitted in separate conversations to assess test-retest consistency, yielding 360 responses overall. Two psychiatrists independently evaluated anonymized outputs against prespecified item-specific reference standards, with clinically important disagreements adjudicated by a third senior psychiatrist. The primary outcome was the proportion of clinically acceptable responses, defined using accuracy, clinical safety, geriatric appropriateness, warning-sign recognition, and triage criteria. Results: Clinically acceptable responses were generated for 78.9% of questions by ChatGPT, 72.2% by Gemini, and 60.0% by Doubao (overall P<0.001). The difference between ChatGPT and Gemini was not statistically significant, whereas both outperformed Doubao. This model ranking remained unchanged under alternative core-safety and more stringent optimal-response definitions. Complete geriatric-specific appropriateness was achieved in 64.4%, 55.6%, and 43.3% of responses, respectively. Major safety errors occurred in 5.6% of ChatGPT responses, 10.0% of Gemini responses, and 16.7% of Doubao responses (raw P = 0.015; FDR-adjusted q=0.023). Performance declined substantially with increasing clinical risk. In exploratory Conclusions: The three models answered many late-life depression questions accurately and safely, but none demonstrated consistently reliable performance across complex or high-risk scenarios. Clinically important weaknesses involved geriatric-specific interpretation, recognition of indirect risk signals, crisis-response completeness, under-triage, and response stability. General-purpose large language models may support selected low-risk educational tasks but should not independently guide emergency triage, medication changes, suicide-risk management, or other safety-critical decisions in older adults.