SynthesisBMJ open2021
How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools.
Synthesis in BMJ open, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 8 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
8 citing papers in PubMed, 1 synthesis or guideline pooled it, 18 citations in OpenAlex.
- How to assess applicability and methodological quality of comparative studies of operative interventions in orthopedic trauma surgery.European journal of trauma and emergency surgery : official publication of the European Trauma Society · 2022Pooled it
- APPRAISE: A Tool for Appraising Potential for Bias in Real-World Evidence Studies on Medication Effectiveness or Safety.Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research · 2025Article
- Observational studies of exposure to tobacco and nicotine products: Best practices for maximizing statistical precision and accuracy.iScience · 2025Review
- Article
- Comparing Effectiveness and Safety of SGLT2 Inhibitors vs DPP-4 Inhibitors in Patients With Type 2 Diabetes and Varying Baseline HbA1c Levels.JAMA internal medicine · 2023Article
- Maternal Multiple Sclerosis and Health Outcomes Among the Children: A Systematic Review.Clinical epidemiology · 2023Review
- Cardiovascular Risks of Hydroxychloroquine vs Methotrexate in Patients With Rheumatoid Arthritis.Journal of the American College of Cardiology · 2022Article
- Risk of bias in non-randomized observational studies assessing the relationship between proton-pump inhibitors and adverse kidney outcomes: a systematic review.Therapeutic advances in gastroenterology · 2022Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors at 8 institutions in 3 countries.
Funding
Abstract
objectiveTo determine whether assessment tools for non-randomised studies (NRS) address critical elements that influence the validity of NRS findings for comparative safety and effectiveness of medications.
designSystematic review and Delphi survey. DATA SOURCES: We searched PubMed, Embase, Google, bibliographies of reviews and websites of influential organisations from inception to November 2019. In parallel, we conducted a Delphi survey among the International Society for Pharmacoepidemiology Comparative Effectiveness Research Special Interest Group to identify key methodological challenges for NRS of medications. We created a framework consisting of the reported methodological challenges to evaluate the selected NRS tools. STUDY SELECTION: Checklists or scales assessing NRS. DATA EXTRACTION: Two reviewers extracted general information and content data related to the prespecified framework.
resultsOf 44 tools reviewed, 48% (n=21) assess multiple NRS designs, while other tools specifically addressed case-control (n=12, 27%) or cohort studies (n=11, 25%) only. Response rate to the Delphi survey was 73% (35 out of 48 content experts), and a consensus was reached in only two rounds. Most tools evaluated methods for selecting study participants (n=43, 98%), although only one addressed selection bias due to depletion of susceptibles (2%). Many tools addressed the measurement of exposure and outcome (n=40, 91%), and measurement and control for confounders (n=40, 91%). Most tools have at least one item/question on design-specific sources of bias (n=40, 91%), but only a few investigate reverse causation (n=8, 18%), detection bias (n=4, 9%), time-related bias (n=3, 7%), lack of new-user design (n=2, 5%) or active comparator design (n=0). Few tools address the appropriateness of statistical analyses (n=15, 34%), methods for assessing internal (n=15, 34%) or external validity (n=11, 25%) and statistical uncertainty in the findings (n=21, 48%). None of the reviewed tools investigated all the methodological domains and subdomains.
conclusionsThe acknowledgement of major design-specific sources of bias (eg, lack of new-user design, lack of active comparator design, time-related bias, depletion of susceptibles, reverse causation) and statistical assessment of internal and external validity is currently not sufficiently addressed in most of the existing tools. These critical elements should be integrated to systematically investigate the validity of NRS on comparative safety and effectiveness of medications. SYSTEMATIC REVIEW PROTOCOL AND REGISTRATION: https://osf.io/es65q.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.