ReviewAnnals of epidemiology2017
Summary measures of agreement and association between many raters' ordinal classifications.
Review in Annals of epidemiology, 2017. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 19 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
19 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Prognostic factors of adjuvant chemotherapy discontinuation among stage III colon cancer patients: A survey of medical oncologists and a systematic review and meta-analysis.Cancer medicine · 2020Pooled it
- Detection of glioma infiltration at the tumor margin using quantitative stimulated Raman scattering histology.Scientific reports · 2021Trial
- Utility of 3-Dimensional Printed Models for Distal Radius Fracture Classification.Hand (New York, N.Y.) · 2026Observational
- Large language models show high clinical safety but differences in completeness for reproductive counselling in women with inflammatory rheumatic and musculoskeletal diseases: a comparative expert evaluation.Rheumatology international · 2026Article
- Surgical Versus Medical Management in Older Adults with Traumatic Brain Injury: A Systematic Review and Meta-analysis.Neurocritical care · 2026Article
- Effect of patient factors and quality of life on ECOG assignment and inter-rater agreement in hospitalized cancer patients.The oncologist · 2026Observational
- Interobserver reproducibility of a hybrid three-tier grading system of papillary nonmuscle invasive urothelial carcinoma: an international Uropathology study.Histopathology · 2025Article
- Some common statistical methods for assessing rater agreement in radiological studies.Acta radiologica (Stockholm, Sweden : 1987) · 2025Review
- Development and preliminary inter-rater reliability of the new PROOF tool to measure fidelity of problem-solving therapy for depression delivered by non-specialists in a low-resource African setting.Global mental health (Cambridge, England) · 2025Article
- Inter- and Intra-Observer Variability of the AMADEUS Tool for Osteochondral Lesions of the Talus.Journal of personalized medicine · 2024Article
- Statistical Methods for the Analysis of Inter-Reader Agreement Among Three or More Readers.Korean journal of radiology · 2024Article
- Bladder ultrasound: evidence of content validity of a checklist for training nurses.Revista brasileira de enfermagem · 2024Article
- Interobserver Variability in the Assessment of Fluorescence Angiography in the Colon.Surgical innovation · 2023Article
- Persistent inter-observer variability of breast density assessment using BI-RADS® 5th edition guidelines.Clinical imaging · 2022Article
- The ESSENCE-Questionnaire in Medical Records Screening for Neurodevelopmental Symptoms/Problems: Utility and Clinical Validity.Neuropsychiatric disease and treatment · 2022Article
- Variability in grading of ductal carcinoma in situ among an international group of pathologists.The journal of pathology. Clinical research · 2021Article
- Wheat Spike Blast Image Classification Using Deep Convolutional Neural Networks.Frontiers in plant science · 2021Article
- Methods of assessing categorical agreement between correlated screening tests in clinical studies.Journal of applied statistics · 2021Article
- Development of a definition and rules for causal attribution of post-colonoscopy bleeding.PloS one · 2020Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
Abstract
purposeInterpretation of screening tests such as mammograms usually require a radiologist's subjective visual assessment of images, often resulting in substantial discrepancies between radiologists' classifications of subjects' test results. In clinical screening studies to assess the strength of agreement between experts, multiple raters are often recruited to assess subjects' test results using an ordinal classification scale. However, using traditional measures of agreement in some studies is challenging because of the presence of many raters, the use of an ordinal classification scale, and unbalanced data.
methodsWe assess and compare the performances of existing measures of agreement and association as well as a newly developed model-based measure of agreement to three large-scale clinical screening studies involving many raters' ordinal classifications. We also conduct a simulation study to demonstrate the key properties of the summary measures.
resultsThe assessment of agreement and association varied according to the choice of summary measure. Some measures were influenced by the underlying prevalence of disease and raters' marginal distributions and/or were limited in use to balanced data sets where every rater classifies every subject. Our simulation study indicated that popular measures of agreement and association are prone to underlying disease prevalence.
conclusionsModel-based measures provide a flexible approach for calculating agreement and association and are robust to missing and unbalanced data as well as the underlying disease prevalence.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.