ArticleBMJ (Clinical research ed.)2016
Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment.
Article in BMJ (Clinical research ed.), 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 7 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
7 citing papers in PubMed, 1 synthesis or guideline pooled it, 45 citations in OpenAlex.
- Effect sizes in ongoing randomized controlled critical care trials.Critical care (London, England) · 2017Pooled it
- Magnitude of effect and sample size justification in trials supporting anti-cancer drug approval by the US Food and Drug Administration.Scientific reports · 2024Article
- Identification of threshold for large (dramatic) effects that would obviate randomized trials is not possible.Journal of clinical epidemiology · 2022Article
- Excess significance and power miscalculations in neurofeedback research.NeuroImage. Clinical · 2022Article
- Late translational research: putting forward a new model for developing new anti-cancer treatments that addresses the needs of patients and society.Molecular oncology · 2019Review
- Smaller clinical trials for decision making; a case study to show p-values are costly.F1000Research · 2018Article
- Reflections on Ethics and Humanity in Pediatric Neurology: the Value of Recognizing Ethical Issues in Common Clinical Practice.Current neurology and neuroscience reports · 2017Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors at 6 institutions in 3 countries.
Funding
Abstract
objectiveTo examine whether a very large effect (VLE; defined as a relative risk of ≤0.2 or ≥5) in a randomised trial could be an empirical marker that subsequent trials are unnecessary.
designMeta-epidemiological assessment of existing published data on randomised trials. DATA SOURCES: Cochrane Database of Systematic Reviews (2010, issue 7) with data on subsequent large trials updated to 2015, issue 12. ELIGIBILITY CRITERIA: All binary outcome forest plots were selected, which contained an index randomised trial with a VLE that was nominally statistically significant (P<0.05), included a subsequent large randomised trial (≥200 events and ≥200 non-events) for validation of the effect, assessed a primary outcome of the review, and was not a subgroup or sensitivity analysis.
resultsOf 3082 reviews yielding 85 002 forest plots, only 44 (0.05%) satisfied the inclusion criteria. Index trials were generally small, with a median sample of 99 (median 14 events). Few index trials were rated at low risk of bias (9 of 44; 20%). The relative risk was closer to the null in the subsequent large trials in 43 of 44 cases. Subsequent large trial data failed to find a statistically significant (P<0.05) effect in the same direction in 19 cases (43%, 95% confidence interval 29% to 58%). Even when the subsequent large trials did find a significant effect in the same direction, the additional primary outcomes in most of these trials would have to be considered before deciding in favour of using the intervention. Subsequent large trial data found a statistically significant effect in the same direction in 19 of 21 cases when the index trial also had a value of P<0.001.
conclusionsThe frequency of VLEs followed by a large trial is vanishingly small, and where they occur they do not appear to be a reliable marker for a benefit that is reproducible and directly actionable. An empirical rule using a VLE in a randomised controlled trial as a marker that further trials are unnecessary would be neither practical nor useful. Caution should be taken when interpreting small studies with very large treatment effects.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.