Evidence map›Paper›PMID 35017250›Full record

SynthesisBMJ open2022

Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength.

Noah A Haber, Emma Clarke-Deelder, Avi Feller, Emily R Smith, Joshua A Salomon, Benjamin MacCormack-Gelles, Elizabeth M Stone, Clara Bolster-Foucault, Jamie R Daw, Laura Anne Hatfield and 14 more

Abstract readSystematic Review
PubMed Full text at PMC Publisher
In one paragraph

Synthesis in BMJ open, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 19 papers, 1 of them a synthesis that pooled it.

0numbers the graph read from it
0cells of the map it votes in
19citing papers in PubMed, 1 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

19 citing papers in PubMed, 1 synthesis or guideline pooled it.

  1. A Systematic Review of Economic Evaluations of COVID-19 Interventions: Considerations of Non-Health Impacts and Distributional Issues.Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research · 2022
    Pooled it
  2. Informed decisions about public health and social measures.Health research policy and systems · 2025
    Review
  3. Review
  4. Review
  5. Article
  6. Article
  7. Article
  8. Article
  9. Article
  10. Article
  11. Policy evaluation during a pandemic.Journal of econometrics · 2023
    Article
  12. Article
  13. Article
  14. Reflections on the COVID-19 pandemic.Journal of paediatrics and child health · 2022
    Article
  15. Article
  16. Article
  17. Review
  18. Article
  19. Article
4 · The record

Corrections and comments

5 · Who and what money

Authors and funding

24 authors.

Noah A HaberMeta Research Innovation Center at Stanford University (METRICS), Stanford University, Stanford, California, USA noahhaber@stanford.edu.ORCID 0000-0002-5672-1769
Emma Clarke-DeelderDepartment of Global Health and Population, Harvard University T H Chan School of Public Health, Boston, Massachusetts, USA.
Avi FellerDepartment of Statistics, Goldman School of Public Policy, University of California Berkeley, Berkeley, California, USA.
Emily R SmithDepartment of Global Health, George Washington University School of Public Health and Health Services, Washington, District of Columbia, USA.
Joshua A SalomonDepartment of Health Policy, Stanford University, Stanford, CA, USA.
Benjamin MacCormack-GellesDepartment of Global Health and Population, Harvard University T H Chan School of Public Health, Boston, Massachusetts, USA.
Elizabeth M StoneDepartment of Health Policy and Management, Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, USA.
Clara Bolster-FoucaultDepartment of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Québec, Canada.
Jamie R DawHealth Policy and Management, Columbia University Mailman School of Public Health, New York, New York, USA.
Laura Anne HatfieldDepartment of Biostatistics, Harvard Medical School, Boston, Massachusetts, USA.ORCID 0000-0003-0366-3929
Carrie E FryDepartment of Health Policy, Vanderbilt University, Nashville, Tennessee, USA.
Christopher B BoyerDepartment of Epidemiology, Harvard University T H Chan School of Public Health, Boston, Massachusetts, USA.
Eli Ben-MichaelInstitute for Quantitative Social Science, Harvard University, Cambridge, MA, USA.
Caroline M JoyceDepartment of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Québec, Canada.
Beth S LinasDepartment of Epidemiology, Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, USA.
Ian SchmidDepartment of Mental Health, Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, USA.
Eric H AuSchool of Public Health, The University of Sydney, Sydney, New South Wales, Australia.
Sarah E WietenMeta Research Innovation Center at Stanford University (METRICS), Stanford University, Stanford, California, USA.
Brooke JarrettDepartment of Epidemiology, Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, USA.ORCID 0000-0003-2966-3521
Cathrine AxforsMeta Research Innovation Center at Stanford University (METRICS), Stanford University, Stanford, California, USA.
Van Thu NguyenMeta Research Innovation Center at Stanford University (METRICS), Stanford University, Stanford, California, USA.
Beth Ann GriffinRAND Corp, Santa Monica, California, USA.
Alyssa BilinskiInterfaculty Initiative in Health Policy, Harvard University Graduate School of Arts and Sciences, Cambridge, Massachusetts, USA.
Elizabeth A StuartDepartment of Mental Health, Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, USA.

Funding

Optimal Methods for Estimating Policy Effect Heterogeneity in Opioid Policy ResearchP50DA046351 · NIDA · RAND CORPORATION · PI Evan David Peet · 2018 to 2026
$20.8M
CVD Epidemiology Training Program in Behavior, the Environment and Global HealthT32HL098048 · NHLBI · HARVARD UNIVERSITY D/B/A HARVARD SCHOOL OF PUBLIC HEALTH · PI Goodarz Danaei, ERIC B RIMM · 2009 to 2026
$6.9M
Integrating data for causal inference in behavioral healthT32MH122357 · NIMH · JOHNS HOPKINS UNIVERSITY · PI Rashelle Jean Musci, Elizabeth A. Stuart · 2020 to 2026
$1.8M
Mental Health Services and Systems Training ProgramT32MH109436 · NIMH · JOHNS HOPKINS UNIVERSITY · PI Matthew Eisenberg, Elizabeth A. Stuart · 2017 to 2026
$1.7M
The Role of an Artificially Intelligent Chatbot in Social Support, Antiretroviral Adherence, and Depressive Symptoms among Young Adults Living with HIV in South AfricaF31MH121128 · NIMH · JOHNS HOPKINS UNIVERSITY · PI JARRETT, BROOKE · 2020 to 2021
$92k
NHLBI NIH HHS T32 HL098048NIDA NIH HHS P50 DA046351NIMH NIH HHS F31 MH121128NIMH NIH HHS T32 MH109436NIMH NIH HHS T32 MH122357
6 · The paper itself

Abstract

introductionAssessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment.

methodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.

resultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Indexed as

COVID-19Cross-Sectional StudiesHealth PolicyHumansResearch DesignSARS-CoV-2COVID-19health policystatistics & research methods

Identifiers

PMID35017250
PMCPMC8753111

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.