ArticleJournal of evaluation in clinical practice2022
High quality (certainty) evidence changes less often than low-quality evidence, but the magnitude of effect size does not systematically differ between studies with low versus high-quality evidence.
Article in Journal of evaluation in clinical practice, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 10 papers, 4 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
10 citing papers in PubMed, 4 syntheses or guidelines pooled it.
- Local bone graft in adolescent idiopathic scoliosis surgery: a systematic review and meta-analysis.Spine deformity · 2026Pooled it
- A Meta-Analytic Review of the Associations of the Big Five Personality Traits with Subjective Poor Sleep Quality and Insomnia, and Meta-Regression of Some Potential Moderators.International journal of behavioral medicine · 2025Pooled it
- Reasons for Downgrading the Certainty of Evidence for Indirectness in Synthesis of Surgical Procedures for Patients With Fractures: A Meta-Research Analysis.Journal of evaluation in clinical practice · 2025Pooled it
- Evidence certainty in neonatology-a meta-epidemiological analysis of Cochrane reviews.European journal of pediatrics · 2025Pooled it
- Conceptualizing the evidence pyramid for use in clinical practice: a narrative literature review.The Journal of the Canadian Chiropractic Association · 2025Article
- Converting Evidence-Based Summary of Findings Evidence Tables Into Decision Analytical, Quality Adjusted Life Years (QALY) and Life Expectancies Metrics: A Tutorial.Journal of evaluation in clinical practice · 2025Article
- What is the probability that higher versus lower quality of evidence represents true effects estimates?Journal of evaluation in clinical practice · 2025Article
- Decision theoretical foundations of clinical practice guidelines: an extension of the ASH thrombophilia guidelines.Blood advances · 2024Article
- Exploring the feasibility of using the ICER Evidence Rating Matrix for Comparative Clinical Effectiveness in assessing treatment benefit and certainty in the clinical evidence on orphan therapies for paediatric indications.Orphanet journal of rare diseases · 2023Article
- High quality (certainty) evidence changes less often than low-quality evidence, but the magnitude of effect size does not systematically differ between studies with low versus high-quality evidence.Journal of evaluation in clinical practice · 2022Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
15 authors.
Funding
Abstract
RATIONALE, AIMS, AND
objectivesIt is generally believed that evidence from low quality of evidence generate inaccurate estimates about treatment effects more often than evidence from high (certainty) quality evidence (CoE). As a result, we would expect that (a) estimates of effects of health interventions initially based on high CoE change less frequently than the effects estimated by lower CoE (b) the estimates of magnitude of effect size differ between high and low CoE. Empirical assessment of these foundational principles of evidence-based medicine has been lacking.
methodsWe reviewed the Cochrane Database of Systematic Reviews from January 2016 through May 2021 for pairs of original and updated reviews for change in CoE assessments based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method. We assessed the difference in effect sizes between the original versus updated reviews as a function of change in CoE, which we report as a ratio of odds ratio (ROR). We compared ROR generated in the studies in which CoE changed from very low/low (VL/L) to moderate/high (M/H) versus M/H to VL/L. Heterogeneity and inconsistency were assessed using the tau and I
resultsFour hundred and nineteen pairs of reviews were included of which 414 (207 × 2) informed the CoE appraisal and 384 (192 × 2) the assessment of effect size. We found that CoE originally appraised as VL/L had 2.1 [95% confidence interval (CI): 1.19-4.12; p = 0.0091] times higher odds to be changed in the future studies than M/H CoE. However, the effect size was not different (p = 1) when CoE changed from VL/L → M/H [ROR = 1.02 (95% CI: 0.74-1.39)] compared with M/H → VL/L (ROR = 1.02 [95% CI: 0.44-2.37]). Similar overlap in aROR between the VL/L → M/H versus M/H → VL/L subgroups was observed [median (IQR): 1.12 (1.07-1.57) vs. 1.21 (1.12-2.43)]. We observed large inconsistency across ROR estimates (I
conclusionsWe found that low-quality evidence changes more often than high CoE. However, the effect size did not systematically differ between the studies with low versus high CoE. The finding that the effect size did not differ between low and high CoE indicate urgent need to refine current EBM critical appraisal methods.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.