ArticleBMJ (Clinical research ed.)2009
Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study.
Article in BMJ (Clinical research ed.), 2009. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 33 papers, 12 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
33 citing papers in PubMed, 12 syntheses or guidelines pooled it.
- Quality appraisal of systematic reviews on high-intensity laser therapy for musculoskeletal pain management: an umbrella review.Lasers in medical science · 2024Pooled it
- The Relationship Between Cognitive Functioning and Symptoms of Depression, Anxiety, and Post-Traumatic Stress Disorder in Adults with a Traumatic Brain Injury: a Meta-Analysis.Neuropsychology review · 2022Pooled it
- Interictal Cognitive Deficits in Migraine Sufferers: A Meta-Analysis.Neuropsychology review · 2022Pooled it
- Associations of Multidomain Interventions With Improvements in Cognition in Mild Cognitive Impairment: A Systematic Review and Meta-analysis.JAMA network open · 2022Pooled it
- The Effectiveness of Fundamental Movement Skill Interventions on Moderate to Vigorous Physical Activity Levels in 5- to 11-Year-Old Children: A Systematic Review and Meta-Analysis.Sports medicine (Auckland, N.Z.) · 2022Pooled it
- Evaluation of various estimators for standardized mean difference in meta-analysis.Statistics in medicine · 2021Pooled it
- Corticosteroids and Cognition: A Meta-Analysis.Neuropsychology review · 2019Pooled it
- A meta-analysis of the effects of antidepressants on cognitive functioning in depressed and non-depressed samples.Neuropsychology review · 2018Pooled it
- Frequency of data extraction errors and methods to increase data extraction quality: a methodological review.BMC medical research methodology · 2017Pooled it
- Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study.BMJ (Clinical research ed.) · 2012Pooled it
- Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study.BMJ (Clinical research ed.) · 2010Pooled it
- Joint lavage for osteoarthritis of the knee.The Cochrane database of systematic reviews · 2010Pooled it
- Evaluation of the replicability of systematic reviews with meta-analyses of the effects of health interventions.Research synthesis methods · 2026Article
- Training and Experience in Study Selection (TESS): study protocol for a pilot randomised trial within a systematic review.HRB open research · 2025Article
- Harm effects in non-registered versus registered randomized controlled trials of medications: a retrospective cohort study of clinical trials.BMC medicine · 2024Article
- Article
- Rethinking clinical study data: why we should respect analysis results as data.Scientific data · 2022Article
- Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study.BMJ (Clinical research ed.) · 2022Article
- A meta-review of transparency and reproducibility-related reporting practices in published meta-analyses on clinical psychological interventions (2000-2020).Behavior research methods · 2022Review
- A new method for testing reproducibility in systematic reviews was developed, but needs more testing.BMC medical research methodology · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
Abstract
objectiveTo study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses.
designObserver agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals.
resultsThe agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review.
conclusionsDisagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.