ArticleSleep2023
Scoring sleep with artificial intelligence enables quantification of sleep stage ambiguity: hypnodensity based on multiple expert scorers and auto-scoring.
Article in Sleep, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 35 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
35 citing papers in PubMed.
- Trial
- Article
- From sleep staging to spindle detection: a case study on end-to-end automated sleep analysis.Scientific reports · 2026Article
- Article
- Automated sleep stage and event detection algorithms using quality-controlled polysomnography annotations.Sleep advances : a journal of the Sleep Research Society · 2026Article
- Beyond Algorithms: The Case for Standardized Reporting in AI Sleep Scoring.Nature and science of sleep · 2026Article
- Temporally Continuous Automated Sleep-Wake Classification Using Deep Learning.medRxiv : the preprint server for health sciences · 2025Article
- Hypnogram and Hypnodensity Analysis of REM Sleep Behaviour Disorder Using Both EEG and HRV-Based Sleep Staging Models.Journal of sleep research · 2025Article
- About Digitalisation and AI, Data Protection, Data Exchange, Data Mining-Legal Constraints/Challenges Concerning Sleep Medicine.Journal of sleep research · 2025Review
- Sleep stage mixing is associated with poor prognosis in early Parkinson's disease.NPJ Parkinson's disease · 2025Article
- Beyond accuracy: a framework for evaluating algorithmic bias and performance, applied to automated sleep scoring.Scientific reports · 2025Article
- Retrospective validation of automatic sleep analysis with grey areas model for human-in-the-loop scoring approach.Journal of sleep research · 2025Article
- A Hierarchical Framework for Selecting Reference Measures for the Analytical Validation of Sensor-Based Digital Health Technologies.Journal of medical Internet research · 2025Article
- Open-Source Algorithm for Automated Vigilance State Classification Using Single-Channel Electroencephalogram in Rodents.Sensors (Basel, Switzerland) · 2025Article
- Validation of the Somnolyzer 24×7 automatic scoring system in children with suspected obstructive sleep apnea.Frontiers in medicine · 2025Article
- Modeling of whole brain sleep electroencephalogram using deep oscillatory neural network.Frontiers in neuroinformatics · 2025Article
- Overnight Sleep Staging Using Chest-Worn Accelerometry.Sensors (Basel, Switzerland) · 2024Article
- NAPping PAnts (NAPPA): An open wearable solution for monitoring Infant's sleeping rhythms, respiration and posture.Heliyon · 2024Article
- Artificial Intelligence Models for the Automation of Standard Diagnostics in Sleep Medicine-A Systematic Review.Bioengineering (Basel, Switzerland) · 2024Review
- Multicentre sleep-stage scoring agreement in the Sleep Revolution project.Journal of sleep research · 2024Article
Corrections and comments
- Commented on by
Authors and funding
9 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
STUDY
objectivesTo quantify the amount of sleep stage ambiguity across expert scorers and to validate a new auto-scoring platform against sleep staging performed by multiple scorers.
methodsWe applied a new auto-scoring system to three datasets containing 95 PSGs scored by 6-12 scorers, to compare sleep stage probabilities (hypnodensity; i.e. the probability of each sleep stage being assigned to a given epoch) as the primary output, as well as a single sleep stage per epoch assigned by hierarchical majority rule.
resultsThe percentage of epochs with 100% agreement across scorers was 46 ± 9%, 38 ± 10% and 32 ± 9% for the datasets with 6, 9, and 12 scorers, respectively. The mean intra-class correlation coefficient between sleep stage probabilities from auto- and manual-scoring was 0.91, representing excellent reliability. Within each dataset, agreement between auto-scoring and consensus manual-scoring was significantly higher than agreement between manual-scoring and consensus manual-scoring (0.78 vs. 0.69; 0.74 vs. 0.67; and 0.75 vs. 0.67; all p < 0.01).
conclusionsAnalysis of scoring performed by multiple scorers reveals that sleep stage ambiguity is the rule rather than the exception. Probabilities of the sleep stages determined by artificial intelligence auto-scoring provide an excellent estimate of this ambiguity. Compared to consensus manual-scoring, sleep staging derived from auto-scoring is for each individual PSG noninferior to manual-scoring meaning that auto-scoring output is ready for interpretation without the need for manual adjustment.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.