Evidence map›Paper›PMID 26858277›Full record

ArticleBMJ (Clinical research ed.)2016

Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey.

Lars G Hemkens, Despina G Contopoulos-Ioannidis, John P A Ioannidis

Erratum issued Registry-linked trialAbstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in BMJ (Clinical research ed.), 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. An erratum has been issued. It is linked to trial NCT04212390 (Personalized Medicine Program on Myelodysplastic Syndromes), which is not on this map. Cited by 73 papers, 13 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
73citing papers in PubMed, 13 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

NCT04212390 completednot on this mapstarted 2020, after this paper: background citation

Personalized Medicine Program on Myelodysplastic Syndromes: Characterization of the Patient's Genome for Clinical Decision Making and Systematic Collection of Real World Data to Improve Quality of Health Care

TypeobservationalSponsorFondazione Italiana Sindromi Mielodisplastiche-ETSRan2020 to 2024Enrolled1,000ConditionsMDS (Myelodysplastic Syndrome)
3 · Its place in the literature

Who cites it

73 citing papers in PubMed, 13 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Non-invasive positive pressure ventilation for acute asthma in children.The Cochrane database of systematic reviews · 2024
    Pooled it
  6. Pooled it
  7. Pooled it
  8. Cancer-related fatigue in patients treated with mistletoe extracts: a systematic review and meta-analysis.Supportive care in cancer : official journal of the Multinational Association of Supportive Care in Cancer · 2022
    Pooled it
  9. Pooled it
  10. Pooled it
  11. Pooled it
  12. Pooled it
  13. Pooled it
  14. Article
  15. Review
  16. Article
  17. Article
  18. Observational
  19. Ethical considerations on the use of big data and artificial intelligence in kidney research from the ERA ethics committee.Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association · 2025
    Review
  20. Article

13 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

5 · Who and what money

Authors and funding

3 authors.

Lars G HemkensStanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland.
Despina G Contopoulos-IoannidisDepartment of Pediatrics, Division of Infectious Diseases, Stanford University School of Medicine, Stanford, California, USA Meta-Research Innovation Center at Stanford (METRICS).
John P A IoannidisStanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA Meta-Research Innovation Center at Stanford (METRICS) Department of Health Research and Policy, Stanford University School of Medicine, Stanford, California, USA Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, USA jioannid@stanford.edu.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

objectiveTo assess differences in estimated treatment effects for mortality between observational studies with routinely collected health data (RCD; that are published before trials are available) and subsequent evidence from randomized controlled trials on the same clinical question.

designMeta-epidemiological survey. DATA SOURCES: PubMed searched up to November 2014.

methodsEligible RCD studies were published up to 2010 that used propensity scores to address confounding bias and reported comparative effects of interventions for mortality. The analysis included only RCD studies conducted before any trial was published on the same topic. The direction of treatment effects, confidence intervals, and effect sizes (odds ratios) were compared between RCD studies and randomized controlled trials. The relative odds ratio (that is, the summary odds ratio of trial(s) divided by the RCD study estimate) and the summary relative odds ratio were calculated across all pairs of RCD studies and trials. A summary relative odds ratio greater than one indicates that RCD studies gave more favorable mortality results.

resultsThe evaluation included 16 eligible RCD studies, and 36 subsequent published randomized controlled trials investigating the same clinical questions (with 17,275 patients and 835 deaths). Trials were published a median of three years after the corresponding RCD study. For five (31%) of the 16 clinical questions, the direction of treatment effects differed between RCD studies and trials. Confidence intervals in nine (56%) RCD studies did not include the RCT effect estimate. Overall, RCD studies showed significantly more favorable mortality estimates by 31% than subsequent trials (summary relative odds ratio 1.31 (95% confidence interval 1.03 to 1.65; I(2)=0%)).

conclusionsStudies of routinely collected health data could give different answers from subsequent randomized controlled trials on the same clinical questions, and may substantially overestimate treatment effects. Caution is needed to prevent misguided clinical decision making.

Indexed as

Observational Studies as TopicRandomized Controlled Trials as TopicHumansMeta-Analysis as TopicOutcome Assessment, Health CarePropensity Score

Identifiers

PMID26858277
PMCPMC4772787

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.