ArticleJournal of medical Internet research2024
Evaluating and Enhancing Large Language Models' Performance in Domain-Specific Medicine: Development and Usability Study With DocOA.
Article in Journal of medical Internet research, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 28 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
28 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines.Journal of the American Medical Informatics Association : JAMIA · 2025Pooled it
- Evaluation of DeepSeek with a RAG-based knowledge base in preoperative assessment for arthroplasty to improve the decision-making accuracy of junior physicians.Arthroplasty (London, England) · 2026Article
- Evaluating the First CE-Marked LLM-Based Chatbot for Evidence-Based Neurology.European journal of neurology · 2026Article
- Image-Quality-Aware Multimodal Artificial Intelligence for Automated Structured OCT Report Generation in Glaucoma Evaluation.Ophthalmology science · 2026Article
- AI in variant analysis: fast track to genetic diagnoses.Human genetics · 2026Review
- Evaluation of ChatGPT-4o's and DeepSeek R1's responses to urological problems: a comparative study.BMC urology · 2026Article
- Authoritative Textbook-Augmented Large Language Models for High-Altitude Public Health Medical Education in the Xizang Autonomous Region: Cross-Sectional Comparative Evaluation Study.Journal of medical Internet research · 2026Article
- The application of large language models in orthopedic postgraduate education: potentials, challenges, and future prospects.Journal of orthopaedic surgery and research · 2026Review
- Evaluating the clinical decision-making performance of large language models in clinically oriented thoracic anatomy scenarios: a comparative evaluation study.BMC medical education · 2026Article
- Gonarthrosis Advisor vs ChatGPT-5: quality and readability of artificial intelligence-generated patient education for knee osteoarthritis.Acta orthopaedica et traumatologica turcica · 2026Article
- A multi-layer retrieval-augmented large language model framework for enhancing hypertension education.Hypertension research : official journal of the Japanese Society of Hypertension · 2026Article
- Grounding large language models in clinical diagnostics.Nature communications · 2026Article
- A Survey on Medical Competence Evaluation Benchmarks for Large Language Models.Health care science · 2026Review
- Comparative analysis of six large language models in perioperative decision support for geriatric patients with multimorbidity: a three-dimensional evaluation framework.BMC anesthesiology · 2026Article
- Assessing the Quality of AI Responses to Patient Concerns About Axial Spondyloarthritis: Delphi-Based Evaluation.JMIR AI · 2026Article
- Assessing the Efficacy of Ortho GPT: A Comparative Study with Medical Students and General LLMs on Orthopedic Examination Questions.Bioengineering (Basel, Switzerland) · 2025Article
- Large Language Model-Enhanced Drug Repositioning Knowledge Extraction via Long Chain-of-Thought: Development and Evaluation Study.JMIR medical informatics · 2025Article
- Glaucoma Detection and Structured OCT Report Generation via a Fine-tuned Multimodal Large Language Model.ArXiv · 2025Article
- Physician Use of Large Language Models: A Quantitative Study Based on Large-Scale Query-Level Data.Journal of medical Internet research · 2025Article
- Performance of Open-Source Large Language Models in Psychiatry: Usability Study Through Comparative Analysis of Non-English Records and English Translations.Journal of medical Internet research · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThe efficacy of large language models (LLMs) in domain-specific medicine, particularly for managing complex diseases such as osteoarthritis (OA), remains largely unexplored.
objectiveThis study focused on evaluating and enhancing the clinical capabilities and explainability of LLMs in specific domains, using OA management as a case study.
methodsA domain-specific benchmark framework was developed to evaluate LLMs across a spectrum from domain-specific knowledge to clinical applications in real-world clinical scenarios. DocOA, a specialized LLM designed for OA management integrating retrieval-augmented generation and instructional prompts, was developed. It can identify the clinical evidence upon which its answers are based through retrieval-augmented generation, thereby demonstrating the explainability of those answers. The study compared the performance of GPT-3.5, GPT-4, and a specialized assistant, DocOA, using objective and human evaluations.
resultsResults showed that general LLMs such as GPT-3.5 and GPT-4 were less effective in the specialized domain of OA management, particularly in providing personalized treatment recommendations. However, DocOA showed significant improvements.
conclusionsThis study introduces a novel benchmark framework that assesses the domain-specific abilities of LLMs in multiple aspects, highlights the limitations of generalized LLMs in clinical contexts, and demonstrates the potential of tailored approaches for developing domain-specific medical LLMs.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.