Evidence map›Paper›PMID 27398997›Full record

SynthesisPloS one2016

Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies.

Matthew J Page, Julian P T Higgins, Gemma Clayton, Jonathan A C Sterne, Asbjørn Hróbjartsson, Jelena Savović

Abstract readSystematic Review
PubMed Full text at PMC Publisher
In one paragraph

Synthesis in PloS one, 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 124 papers, 37 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
124citing papers in PubMed, 37 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

124 citing papers in PubMed, 37 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Pooled it
  16. Pooled it
  17. Risk of bias and reporting practices in studies comparing VOJournal of sport and health science · 2022
    Pooled it
  18. Pooled it
  19. Pooled it
  20. Pooled it

64 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

6 authors.

Matthew J PageSchool of Social and Community Medicine, University of Bristol, Bristol, United Kingdom.
Julian P T HigginsSchool of Social and Community Medicine, University of Bristol, Bristol, United Kingdom.
Gemma ClaytonSchool of Social and Community Medicine, University of Bristol, Bristol, United Kingdom.
Jonathan A C SterneSchool of Social and Community Medicine, University of Bristol, Bristol, United Kingdom.
Asbjørn HróbjartssonCenter for Evidence-Based Medicine, University of Southern Denmark & Odense University Hospital, Odense, Denmark.
Jelena SavovićSchool of Social and Community Medicine, University of Bristol, Bristol, United Kingdom.

Funding

Medical Research Council G0701659Medical Research Council MR/K025643/1Medical Research Council MR/L004933/1Medical Research Council MR/L004933/2
6 · The paper itself

Abstract

objectiveTo synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials.

designSystematic review of meta-epidemiological studies.

methodsWe retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic.

resultsWe included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear.

conclusionsCertain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.

Indexed as

Epidemiologic StudiesBiasHumansRandomized Controlled Trials as Topic

Identifiers

PMID27398997
PMCPMC4939945

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.