ArticleJournal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine2021
Interrater sleep stage scoring reliability between manual scoring from two European sleep centers and automatic scoring performed by the artificial intelligence-based Stanford-STAGES algorithm.
Article in Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 27 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
27 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Machine learning models including patient-reported outcome data in oncology: a systematic literature review and analysis of their reporting quality.Journal of patient-reported outcomes · 2024Pooled it
- Evaluation of consensus sleep stage scoring of dysregulated sleep in Parkinson's disease.Sleep medicine · 2023Trial
- Fully-automated sleep staging for Parkinson's disease and isolated REM sleep behavior disorder.NPJ digital medicine · 2026Article
- Article
- Article
- YASA automated sleep staging performance across seven nights of normal sleep and sleep restriction.Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine · 2026Article
- Automated sleep stage and event detection algorithms using quality-controlled polysomnography annotations.Sleep advances : a journal of the Sleep Research Society · 2026Article
- Beyond Algorithms: The Case for Standardized Reporting in AI Sleep Scoring.Nature and science of sleep · 2026Article
- Article
- Temporal progression of sleep electroencephalography features in isolated rapid eye movement sleep behaviour disorder.Journal of sleep research · 2025Article
- Sleep as the Foundation of Brain Health.Seminars in neurology · 2025Review
- Identifying time-resolved features of nocturnal sleep characteristics of narcolepsy using machine learning.Journal of sleep research · 2024Article
- Semi automatic quantification of REM sleep without atonia in natural sleep environment.NPJ digital medicine · 2024Article
- Machine Learning Predicts Phenoconversion from Polysomnography in Isolated REM Sleep Behavior Disorder.Brain sciences · 2024Article
- Strengths, weaknesses, opportunities, and threats of using AI-enabled technology in sleep medicine: a commentary.Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine · 2024Review
- Artificial Intelligence Models for the Automation of Standard Diagnostics in Sleep Medicine-A Systematic Review.Bioengineering (Basel, Switzerland) · 2024Review
- Article
- DynamicSleepNet: a multi-exit neural network with adaptive inference time for sleep stage classification.Frontiers in physiology · 2023Article
- Overview of the hypnodensity approach to scoring sleep for polysomnography and home sleep testing.Frontiers in sleep · 2023Review
- Enhanced sleep staging with artificial intelligence: a validation study of new software for sleep scoring.Frontiers in artificial intelligence · 2023Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
STUDY
objectivesThe objective of this study was to evaluate interrater reliability between manual sleep stage scoring performed in 2 European sleep centers and automatic sleep stage scoring performed by the previously validated artificial intelligence-based Stanford-STAGES algorithm.
methodsFull night polysomnographies of 1,066 participants were included. Sleep stages were manually scored in Berlin and Innsbruck sleep centers and automatically scored with the Stanford-STAGES algorithm. For each participant, we compared (1) Innsbruck to Berlin scorings (INN vs BER); (2) Innsbruck to automatic scorings (INN vs AUTO); (3) Berlin to automatic scorings (BER vs AUTO); (4) epochs where scorers from Innsbruck and Berlin had consensus to automatic scoring (CONS vs AUTO); and (5) both Innsbruck and Berlin manual scorings (MAN) to the automatic ones (MAN vs AUTO). Interrater reliability was evaluated with several measures, including overall and sleep stage-specific Cohen's κ.
resultsOverall agreement across participants was substantial for INN vs BER (κ = 0.66 ± 0.13), INN vs AUTO (κ = 0.68 ± 0.14), CONS vs AUTO (κ = 0.73 ± 0.14), and MAN vs AUTO (κ = 0.61 ± 0.14), and moderate for BER vs AUTO (κ = 0.55 ± 0.15). Human scorers had the highest disagreement for N1 sleep (κ
conclusionsInterrater reliability for sleep stage scoring between human scorers was in line with previous findings, and the algorithm achieved an overall substantial agreement with manual scoring. In this cohort, the Stanford-STAGES algorithm showed similar performances to the ones achieved in the original study, suggesting that it is generalizable to new cohorts. Before its integration in clinical practice, future independent studies should further evaluate it in other cohorts.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.