ArticlePeerJ2019
We need to talk about reliability: making better use of test-retest studies for study design and interpretation.
Article in PeerJ, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 108 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
108 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Reliability of Temporal Summation of Pain in Healthy and Clinical Populations: A Systematic Review and Meta-Analysis.European journal of pain (London, England) · 2025Pooled it
- Tests employed in the psychometric validation of the Insulin Treatment Appraisal Scale (ITAS) in T2DM patients; a systematic review of the literature.Journal of patient-reported outcomes · 2024Pooled it
- Submaximal Fitness Test in Team Sports: A Systematic Review and Meta-Analysis of Exercise Heart Rate Measurement Properties.Sports medicine - open · 2023Pooled it
- Outcome assessment of a complex mental health intervention in the workplace. Results from the MENTUPP pilot study.International archives of occupational and environmental health · 2023Trial
- Effects of Eccentric-Oriented Strength Training on Return to Sport Criteria in Late-Stage Anterior Cruciate Ligament (ACL)-Reconstructed Professional Team Sport Players.Medicina (Kaunas, Lithuania) · 2023Trial
- Sniffer dogs performance is stable over time in detecting COVID-19 positive samples and agrees with the rapid antigen test in the field.Scientific reports · 2023Trial
- Co-clinical FDG-PET radiomic signature in predicting response to neoadjuvant chemotherapy in triple-negative breast cancer.European journal of nuclear medicine and molecular imaging · 2022Trial
- Agreement among common core temperature indices in defining critical environmental limits for human thermoregulation: a secondary analysis.American journal of physiology. Regulatory, integrative and comparative physiology · 2026Article
- Dissociable effects of schema congruency on item recognition and source memory across levels of expectedness and surprise.Memory & cognition · 2026Article
- Continuous assessment of the sweat rate-lactate content association during acetylcholine-induced sweating: an exploratory study.European journal of applied physiology · 2026Article
- Inter- and intra-site reliability of forearm sweating induced by pilocarpine iontophoresis.The journal of physiological sciences : JPS · 2026Article
- Deep learning single-cell analysis for cytologic evaluation of oral potentially malignant disorders.Scientific reports · 2026Article
- An Indoor Accessibility Assessment Framework Based on Multimodal Sensing and Explainable Machine Learning: A Case Study of a Tactile Museum for People with Visual Impairments.Sensors (Basel, Switzerland) · 2026Article
- Excess EEG beta-band oscillations in Dup15q syndrome correlate with clinical severity.Translational psychiatry · 2026Article
- Exploring patient-reported outcomes after lacrimal surgery in a large monocentric cohort: cross-cultural adaptation of the Italian Lac-Q.Head & face medicine · 2026Article
- Precision phenotyping of elevated arousal in posttraumatic stress disorder: implications for personalized GABAergic pharmacotherapy.Frontiers in psychiatry · 2026Review
- Validation and cross-cultural adaptation of an Arabic version of the chronic pain self-efficacy scale.Hong Kong physiotherapy journal : official publication of the Hong Kong Physiotherapy Association Limited = Wu li chih liao · 2025Article
- On the Unreliability of Test-Retest Reliability.Applied psychological measurement · 2025Article
- Large-scale experimental investigation of the reliability of confidence measures.Communications psychology · 2025Article
- Effect of a consistent reconstruction algorithm on inter-scanner reproducibility in diffusion MRI.Medical physics · 2025Article
48 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
1 author.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Neuroimaging, in addition to many other fields of clinical research, is both time-consuming and expensive, and recruitable patients can be scarce. These constraints limit the possibility of large-sample experimental designs, and often lead to statistically underpowered studies. This problem is exacerbated by the use of outcome measures whose accuracy is sometimes insufficient to answer the scientific questions posed. Reliability is usually assessed in validation studies using healthy participants, however these results are often not easily applicable to clinical studies examining different populations. I present a new method and tools for using summary statistics from previously published test-retest studies to approximate the reliability of outcomes in new samples. In this way, the feasibility of a new study can be assessed during planning stages, and before collecting any new data. An R package called relfeas also accompanies this article for performing these calculations. In summary, these methods and tools will allow researchers to avoid performing costly studies which are, by virtue of their design, unlikely to yield informative conclusions.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.