ArticleStatistics in medicine2015
Measures of agreement between many raters for ordinal classifications.
Article in Statistics in medicine, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT04726306 (A Comparative Analysis of Diagnostic Precision and Accuracy Using the TEED and OGS Clinical Grading Systems for Symptomatic Eustachian Tube Dysfunction and Middle Ear Barotrauma), which is not on this map. Cited by 38 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
A Comparative Analysis of Diagnostic Precision and Accuracy Using the TEED and OGS Clinical Grading Systems for Symptomatic Eustachian Tube Dysfunction and Middle Ear Barotrauma: A Validation Study
Who cites it
38 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Prognostic factors of adjuvant chemotherapy discontinuation among stage III colon cancer patients: A survey of medical oncologists and a systematic review and meta-analysis.Cancer medicine · 2020Pooled it
- Detection of glioma infiltration at the tumor margin using quantitative stimulated Raman scattering histology.Scientific reports · 2021Trial
- AI-Assisted Compressed Sensing for Optimized Inner Ear Imaging: a Multi-Rater MRI Evaluation of CSAI T2-DRIVE.Clinical neuroradiology · 2026Article
- [Comparison of missing data handling methods for ACNan fang yi ke da xue xue bao = Journal of Southern Medical University · 2026Article
- Prescription rights for psychologists - exploring professionals' perspective in Poland.Scientific reports · 2025Article
- Bayesian Nonparametric Models for Multiple Raters: A General Statistical Framework.Psychometrika · 2025Article
- Accuracy, Clarity, and Comprehensiveness of ChatGPT Outputs for Commonly Asked Questions About Living Kidney Donation.Clinical transplantation · 2025Article
- Assessing GPT and DeepL for terminology translation in the medical domain: A comparative study on the human phenotype ontology.BMC medical informatics and decision making · 2025Article
- Applying language models for suicide prevention: evaluating news article adherence to WHO reporting guidelines.Npj mental health research · 2025Article
- Expected impact of MRI-targeted biopsy interreader variability among uropathologists on ProScreen prostate cancer screening trial: a pre-trial validation study.World journal of urology · 2024Article
- Optimization of Postprocessing parameters for abdominal Forensic CT scans.Forensic science international. Synergy · 2024Article
- Inter-rater reliability of the infectious disease modeling reproducibility checklist (IDMRC) as applied to COVID-19 computational modeling research.BMC infectious diseases · 2023Article
- Prospective Correlation of Magnetic Resonance Tumor Regression Grade With Pathologic Outcomes in Total Neoadjuvant Therapy for Rectal Adenocarcinoma.Journal of clinical oncology : official journal of the American Society of Clinical Oncology · 2023Article
- Article
- Inter-rater reliability of the Infectious Disease Modeling Reproducibility Checklist (IDMRC) as applied to COVID-19 computational modeling research.medRxiv : the preprint server for health sciences · 2023Article
- Ethics Consultation in Surgical Specialties.HEC forum : an interdisciplinary journal on hospitals' ethical and legal issues · 2022Article
- Inter and intra-observer agreement of arterial wall contrast-enhanced ultrasonography in giant cell arteritis.Frontiers in medicine · 2022Article
- The ESSENCE-Questionnaire in Medical Records Screening for Neurodevelopmental Symptoms/Problems: Utility and Clinical Validity.Neuropsychiatric disease and treatment · 2022Article
- Assessment of variability in motor grading and patient-reported outcome reporting: a multi-specialty, multi-national survey.Acta neurochirurgica · 2021Article
- Variability in grading of ductal carcinoma in situ among an international group of pathologists.The journal of pathology. Clinical research · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
2 authors.
Funding
Abstract
Screening and diagnostic procedures often require a physician's subjective interpretation of a patient's test result using an ordered categorical scale to define the patient's disease severity. Because of wide variability observed between physicians' ratings, many large-scale studies have been conducted to quantify agreement between multiple experts' ordinal classifications in common diagnostic procedures such as mammography. However, very few statistical approaches are available to assess agreement in these large-scale settings. Many existing summary measures of agreement rely on extensions of Cohen's kappa. These are prone to prevalence and marginal distribution issues, become increasingly complex for more than three experts, or are not easily implemented. Here we propose a model-based approach to assess agreement in large-scale studies based upon a framework of ordinal generalized linear mixed models. A summary measure of agreement is proposed for multiple experts assessing the same sample of patients' test results according to an ordered categorical scale. This measure avoids some of the key flaws associated with Cohen's kappa and its extensions. Simulation studies are conducted to demonstrate the validity of the approach with comparison with commonly used agreement measures. The proposed methods are easily implemented using the software package R and are applied to two large-scale cancer agreement studies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.