ArticleJournal of medical Internet research2017
Supervised Machine Learning Algorithms Can Classify Open-Text Feedback of Doctor Performance With Human-Level Accuracy.
Article in Journal of medical Internet research, 2017. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 16 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
16 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Applying natural language processing and machine learning techniques to patient experience feedback: a systematic review.BMJ health & care informatics · 2021Pooled it
- Understanding and applying machine learning in differentiating abdominal IgA vasculitis without purpura from appendicitis.Pediatric research · 2026Article
- Estimation of sexual dimorphism of adult human mandibles of South Indian origin using non-metric parameters and machine learning classification algorithms.Scientific reports · 2025Article
- Leveraging Narrative Feedback in Programmatic Assessment: The Potential of Automated Text Analysis to Support Coaching and Decision-Making in Programmatic Assessment.Advances in medical education and practice · 2024Article
- A multiphase study protocol of identifying, and predicting cancer-related symptom clusters: applying a mixed-method design and machine learning algorithms.Frontiers in digital health · 2024Article
- Enhanced Surgical Decision-Making Tools in Breast Cancer: Predicting 2-Year Postoperative Physical, Sexual, and Psychosocial Well-Being following Mastectomy and Breast Reconstruction (INSPiRED 004).Annals of surgical oncology · 2023Article
- Changes in Doctor-Patient Relationships in China during COVID-19: A Text Mining Analysis.International journal of environmental research and public health · 2022Article
- Co-designing new tools for collecting, analysing and presenting patient experience data in NHS services: working in partnership with patients and carers.Research involvement and engagement · 2021Article
- Maximizing the Potential of Patient-Reported Assessments by Using the Open-Source Concerto Platform With Computerized Adaptive Testing and Machine Learning.Journal of medical Internet research · 2020Article
- Artificial Intelligence Versus Clinicians in Disease Diagnosis: Systematic Review.JMIR medical informatics · 2019Review
- Development of Predictive Models in Patients with Epiphora Using Lacrimal Scintigraphy and Machine Learning.Nuclear medicine and molecular imaging · 2019Article
- Machine learning in medicine: a practical introduction.BMC medical research methodology · 2019Article
- Clinical Value of Predicting Individual Treatment Effects for Intensive Blood Pressure Therapy.Circulation. Cardiovascular quality and outcomes · 2019Article
- Issues in Building a Nursing Home Syndromic Surveillance System with Textmining: Longitudinal Observational Study.JMIR public health and surveillance · 2018Article
- Analyzing Medication Error Reports in Clinical Settings: An Automated Pipeline Approach.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2018Article
- Investigating the meaning of 'good' or 'very good' patient evaluations of care in English general practice: a mixed methods study.BMJ open · 2017Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
Abstract
backgroundMachine learning techniques may be an effective and efficient way to classify open-text reports on doctor's activity for the purposes of quality assurance, safety, and continuing professional development.
objectiveThe objective of the study was to evaluate the accuracy of machine learning algorithms trained to classify open-text reports of doctor performance and to assess the potential for classifications to identify significant differences in doctors' professional performance in the United Kingdom.
methodsWe used 1636 open-text comments (34,283 words) relating to the performance of 548 doctors collected from a survey of clinicians' colleagues using the General Medical Council Colleague Questionnaire (GMC-CQ). We coded 77.75% (1272/1636) of the comments into 5 global themes (innovation, interpersonal skills, popularity, professionalism, and respect) using a qualitative framework. We trained 8 machine learning algorithms to classify comments and assessed their performance using several training samples. We evaluated doctor performance using the GMC-CQ and compared scores between doctors with different classifications using t tests.
resultsIndividual algorithm performance was high (range F score=.68 to .83). Interrater agreement between the algorithms and the human coder was highest for codes relating to "popular" (recall=.97), "innovator" (recall=.98), and "respected" (recall=.87) codes and was lower for the "interpersonal" (recall=.80) and "professional" (recall=.82) codes. A 10-fold cross-validation demonstrated similar performance in each analysis. When combined together into an ensemble of multiple algorithms, mean human-computer interrater agreement was .88. Comments that were classified as "respected," "professional," and "interpersonal" related to higher doctor scores on the GMC-CQ compared with comments that were not classified (P<.05). Scores did not vary between doctors who were rated as popular or innovative and those who were not rated at all (P>.05).
conclusionsMachine learning algorithms can classify open-text feedback of doctor performance into multiple themes derived by human raters with high performance. Colleague open-text comments that signal respect, professionalism, and being interpersonal may be key indicators of doctor's performance.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.