ArticleNeuroImage2021
Trial and error: A hierarchical modeling approach to test-retest reliability.
Article in NeuroImage, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 33 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
33 citing papers in PubMed.
- Habenula Alterations in Resting-State Functional Connectivity Among Autistic Individuals.Biological psychiatry. Cognitive neuroscience and neuroimaging · 2026Article
- Go figure: transparency in neuroscience images preserves context and clarifies interpretation.Nature methods · 2026Review
- Neurobiological Signatures of Dyadic Transmission of Fear Extinction in Adolescent Trauma Exposure and Posttraumatic Stress.Biological psychiatry global open science · 2026Article
- Age-Related Differences in Naming Latency Variability.Journal of cognition · 2026Article
- Dense Phenotyping of Human Brain Network Organization Using Precision fMRI.Annual review of psychology · 2026Review
- Implementation of support vector machines to classify abnormal neuronal response during emotion regulation at an individual level in patients with newly diagnosed bipolar disorder - and its association with subsequent functional changes and mood episodes.Psychological medicine · 2025Article
- Using Machine Learning to Determine a Functional Classifier of Retaliation and Its Association With Aggression.JAACAP open · 2025Article
- Trait state occasion (TSO) modeling of event-related potentials (ERPs).Biological psychology · 2025Article
- Complementary benefits of multivariate and hierarchical models for identifying individual differences in cognitive control.Imaging neuroscience (Cambridge, Mass.) · 2025Article
- Group-to-individual generalizability and individual-level inferences in cognitive neuroscience.Neuroscience and biobehavioral reviews · 2025Review
- Improving accuracy and precision of heritability estimation in twin studies through hierarchical modeling: reassessing the measurement error assumption.Frontiers in genetics · 2025Article
- Complementary benefits of multivariate and hierarchical models for identifying individual differences in cognitive control.bioRxiv : the preprint server for biology · 2024Article
- Reliability of the serial reaction time task: If at first you don't succeed, try, try, try again.Quarterly journal of experimental psychology (2006) · 2024Article
- The value of error-correcting responses for cognitive assessment in games.Scientific reports · 2024Article
- Reliable affordances: A generative modeling approach for test-retest reliability of the affordances task.Behavior research methods · 2024Article
- A comparison between different variants of the spatial Stroop task: The influence of analytic flexibility on Stroop effect estimates and reliability.Behavior research methods · 2024Article
- Are Faster Participants Always Faster? Assessing Reliability of Participants' Mean Response Speed in Picture Naming.Journal of cognition · 2024Article
- A precision neuroscience approach to estimating reliability of neural responses during emotion processing: Implications for task-fMRI.NeuroImage · 2024Article
- BOLD Response is more than just magnitude: Improving detection sensitivity through capturing hemodynamic profiles.NeuroImage · 2023Article
- Improving the Reliability of Cognitive Task Measures: A Narrative Review.Biological psychiatry. Cognitive neuroscience and neuroimaging · 2023Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
Abstract
The concept of test-retest reliability indexes the consistency of a measurement across time. High reliability is critical for any scientific study, but specifically for the study of individual differences. Evidence of poor reliability of commonly used behavioral and functional neuroimaging tasks is mounting. Reports on low reliability of task-based fMRI have called into question the adequacy of using even the most common, well-characterized cognitive tasks with robust population-level effects, to measure individual differences. Here, we lay out a hierarchical framework that estimates reliability as a correlation divorced from trial-level variability, and show that reliability tends to be underestimated under the conventional intraclass correlation framework through summary statistics based on condition-level modeling. In addition, we examine how reliability estimation between the two statistical frameworks diverges and assess how different factors (e.g., trial and subject sample sizes, relative magnitude of cross-trial variability) impact reliability estimates. As empirical data indicate that cross-trial variability is large in most tasks, this work highlights that a large number of trials (e.g., greater than 100) may be required to achieve precise reliability estimates. We reference the tools TRR and 3dLMEr for the community to apply trial-level models to behavior and neuroimaging data and discuss how to make these new measurements most useful for future studies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.