ArticleStatistics in medicine2018
Performance of intraclass correlation coefficient (ICC) as a reliability index under various distributions in scale reliability studies.
Article in Statistics in medicine, 2018. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 84 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
84 citing papers in PubMed.
- Comparison of case-based and lecture-based learning in dental fluorosis diagnostic ability with visual analog scale assessment.BMC medical education · 2024Trial
- Interrater reliability of photographic assessment of thyroid eye disease using the VISA classification.International ophthalmology · 2024Trial
- Shared Decision Making for Syncope in the Emergency Department: A Randomized Controlled Feasibility Trial.Academic emergency medicine : official journal of the Society for Academic Emergency Medicine · 2020Trial
- Pediatric Auditory-Perceptual Evaluation of Voice Part 2: A Protocol.Journal of speech, language, and hearing research : JSLHR · 2026Article
- What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education.JMIR medical education · 2026Article
- Ultrasound gap measurement after acute Achilles rupture is reliable overall but uncertain near a 5-mm decision threshold.Skeletal radiology · 2026Article
- Development and psychometric evaluation of a workplace psychological hazards scale for operating room nurses.BMC nursing · 2026Article
- Impact of Different Energy Levels of Virtual Monoenergetic Reconstructions on Radiomic Features Stability in Organic Phantom Imaging Using Photon-Counting CT.Tomography (Ann Arbor, Mich.) · 2026Article
- Evaluating policy guidelines for the prevention and management of anaemia in pregnancy in Africa: a systematic scoping review of practices and recommendations.Tropical medicine and health · 2026Article
- Intra-Rater and Test-Retest Reliability of Kinovea for the Kinematic Analysis of Squatting in Healthy Active Women.Sensors (Basel, Switzerland) · 2026Article
- Assessing post-harvest loss data reporting methods for tomato in least developed countries.Scientific reports · 2026Article
- Deep learning-based infrared thermography reveals reproducible uniform and individual thermoregulatory responses during running.Scientific reports · 2026Article
- Article
- Test-retest reliability of the gaze index for sign-tracking and goal-tracking.Behavior research methods · 2026Article
- Interocular symmetry of retinal and choroidal OCT parameters in precapillary pulmonary hypertension: a brief research report.Frontiers in medicine · 2026Article
- Article
- Interpretable ADC-based radiomics models for differentiating hepatocellular carcinoma and intrahepatic cholangiocarcinoma.Frontiers in oncology · 2026Article
- Radiomics and clinical data predict pseudoprogression after radiotherapy in high-grade glioma.Frontiers in oncology · 2026Article
- Improving the Accuracy and Reliability of Ratings on the Hamilton Depression Rating Scale via a Video-Based Training Program.Acta psychiatrica Scandinavica · 2025Article
- Automated Field of View Prescription for Whole-body Magnetic Resonance Imaging Using Deep Learning Based Body Region Segmentations.Investigative radiology · 2025Article
24 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Many published scale validation studies determine inter-rater reliability using the intra-class correlation coefficient (ICC). However, the use of this statistic must consider its advantages, limitations, and applicability. This paper evaluates how interaction of subject distribution, sample size, and levels of rater disagreement affects ICC and provides an approach for obtaining relevant ICC estimates under suboptimal conditions. Simulation results suggest that for a fixed number of subjects, ICC from the convex distribution is smaller than ICC for the uniform distribution, which in turn is smaller than ICC for the concave distribution. The variance component estimates also show that the dissimilarity of ICC among distributions is attributed to the study design (ie, distribution of subjects) component of subject variability and not the scale quality component of rater error variability. The dependency of ICC on the distribution of subjects makes it difficult to compare results across reliability studies. Hence, it is proposed that reliability studies should be designed using a uniform distribution of subjects because of the standardization it provides for representing objective disagreement. In the absence of uniform distribution, a sampling method is proposed to reduce the non-uniformity. In addition, as expected, high levels of disagreement result in low ICC, and when the type of distribution is fixed, any increase in the number of subjects beyond a moderately large specification such as n = 80 does not have a major impact on ICC.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.