Evidence map›Paper›PMID 37491242›Full record

ArticleImplementation science : IS2023

Ten years of implementation outcomes research: a scoping review.

Enola K Proctor, Alicia C Bunger, Rebecca Lengnick-Hall, Donald R Gerke, Jared K Martin, Rebecca J Phillips, Julia C Swanson

Erratum issuedAbstract readScoping Review
PubMed Full text at PMC Publisher
In one paragraph

Article in Implementation science : IS, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. An erratum has been issued. Cited by 283 papers, 7 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
283citing papers in PubMed, 7 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

283 citing papers in PubMed, 7 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Application of the Knowledge-to-Action framework in translating nutrition and dietetic knowledge into practice: A systematic review.Nutrition in clinical practice : official publication of the American Society for Parenteral and Enteral Nutrition · 2025
    Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Trial
  9. Trial
  10. Trial
  11. Trial
  12. Trial
  13. Trial
  14. Trial
  15. Trial
  16. Trial
  17. Trial
  18. Trial
  19. Trial
  20. Trial

223 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

5 · Who and what money

Authors and funding

7 authors.

Enola K ProctorThe Brown School, Shanti Khinduka Distinguished Professor Emerita, Washington University in St. Louis, St. Louis, USA. ekp@wustl.edu.ORCID http://orcid.org/0000-0001-5422-6003
Alicia C BungerCollege of Social Work, The Ohio State University, Columbus, OH, USA.
Rebecca Lengnick-HallThe Brown School, Washington University in St. Louis, St. Louis, USA.
Donald R GerkeDepartment of Social Work, College of Arts and Sciences, University of Alabama at Birmingham, Birmingham, USA.
Jared K MartinCollege of Education & Human Ecology, The Ohio State University, Columbus, OH, USA.
Rebecca J PhillipsCollege of Liberal Arts & Sciences, Western Oregon University, Monmouth, OR, USA.
Julia C SwansonDepartment of Psychiatry, Washington University School of Medicine, St. Louis, MO, USA.

Funding

National Survey of State Policy Decision-making and SimulationP50MH113662 · NIMH · NEW YORK UNIVERSITY SCHOOL OF MEDICINE · PI HOAGWOOD, KIMBERLY E., MCKAY, MARY MCKERNAN · 2019 to 2022
$6.9M
IMPLEMENTATION RESEARCH INSTITUTE FOR MENTAL HEALTH SERVICES IRI-MHSR25MH080916 · NIMH · WASHINGTON UNIVERSITY · PI Byron James Powell · 2009 to 2026
$3.7M
Collaborating to Implement Cross-System Interventions in Child Welfare and Substance UseR34DA046913 · NIDA · OHIO STATE UNIVERSITY · PI BUNGER, ALICIA C · 2019 to 2021
$716k
NIDA NIH HHS R34 DA046913NIMH NIH HHS P50 MH113662NIMH NIH HHS R25 MH080916
6 · The paper itself

Abstract

backgroundProctor and colleagues' 2011 paper proposed a taxonomy of eight implementation outcomes and challenged the field to address a research agenda focused on conceptualization, measurement, and theory building. Ten years later, this paper maps the field's progress in implementation outcomes research. This scoping review describes how each implementation outcome has been studied, research designs and methods used, and the contexts and settings represented in the current literature. We also describe the role of implementation outcomes in relation to implementation strategies and other outcomes.

methodsArksey and O'Malley's framework for conducting scoping reviews guided our methods. Using forward citation tracing, we identified all literature citing the 2011 paper. We conducted our search in the Web of Science (WOS) database and added citation alerts sent to the first author from the publisher for a 6-month period coinciding with the WOS citation search. This produced 1346 titles and abstracts. Initial abstract screening yielded 480 manuscripts, and full-text review yielded 400 manuscripts that met inclusion criteria (empirical assessment of at least one implementation outcome).

resultsSlightly more than half (52.1%) of included manuscripts examined acceptability. Fidelity (39.3%), feasibility (38.6%), adoption (26.5%), and appropriateness (21.8%) were also commonly examined. Penetration (16.0%), sustainability (15.8%), and cost (7.8%) were less frequently examined. Thirty-two manuscripts examined implementation outcomes not included in the original taxonomy. Most studies took place in healthcare (45.8%) or behavioral health (22.5%) organizations. Two-thirds used observational designs. We found little evidence of progress in testing the relationships between implementation strategies and implementation outcomes, leaving us ill-prepared to know how to achieve implementation success. Moreover, few studies tested the impact of implementation outcomes on other important outcome types, such as service systems and improved individual or population health.

conclusionsOur review presents a comprehensive snapshot of the research questions being addressed by existing implementation outcomes literature and reveals the need for rigorous, analytic research and tests of strategies for attaining implementation outcomes in the next 10 years of outcomes research.

Indexed as

Delivery of Health CareOutcome Assessment, Health CareHumansAcceptabilityAdoptionAppropriatenessFeasibilityFidelityImplementation costImplementation outcomePenetrationSustainability

Identifiers

PMID37491242
PMCPMC10367273

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.