Evidence map›Paper›PMID 33355350›Full record

ArticleJournal of the American Medical Informatics Association : JAMIA2021

A framework for making predictive models useful in practice.

Kenneth Jung, Sehj Kashyap, Anand Avati, Stephanie Harman, Heather Shaw, Ron Li, Margaret Smith, Kenny Shum, Jacob Javitz, Yohan Vetteth and 3 more

Registry-linked trialOpen access · bronzeAbstract read
In one paragraph

Article in Journal of the American Medical Informatics Association : JAMIA, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT07167173 (Predictive Model for Multidrug Resistance in Patients Admitted to the Emergency Department With Sepsis), which is not on this map. Cited by 39 papers, 2 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
39citing papers in PubMed, 2 pooled it
4.5field-weighted citation impact, top 5% of its field
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

NCT07167173 not yet recruitingnot on this mapstarted 2025, after this paper: background citation

Predictive Model for Multidrug Resistance in Patients Admitted to the Emergency Department With Sepsis

TypeobservationalSponsorHospital Italiano de Buenos AiresRan2025 to 2025Enrolled10,000ConditionsSepsis, Septic Shock
3 · Its place in the literature

Who cites it

39 citing papers in PubMed, 2 syntheses or guidelines pooled it, 76 citations in OpenAlex.

  1. Pooled it
  2. Pooled it
  3. Loss function influence on hyperparameter optimization for observational healthcare prediction models.Journal of the American Medical Informatics Association : JAMIA · 2026
    Article
  4. Article
  5. Article
  6. Observational
  7. Review
  8. Article
  9. Article
  10. Racial/ethnic discrimination and racial trauma: Concurrent evaluation among Black adults who smoke in the United States.Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors · 2025
    Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Developing a Research Center for Artificial Intelligence in Medicine.Mayo Clinic proceedings. Digital health · 2024
    Article
  16. Article
  17. Ensuring useful adoption of generative artificial intelligence in healthcare.Journal of the American Medical Informatics Association : JAMIA · 2024
    Review
  18. Article
  19. Artificial Intelligence in Geriatrics: Riding the Inevitable Tide of Promise, Challenges, and Considerations.The journals of gerontology. Series A, Biological sciences and medical sciences · 2024
    Article
  20. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

13 authors at 3 institutions in 1 country.

Kenneth JungStanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA.
Sehj KashyapStanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA.
Anand AvatiDepartment of Computer Science, School of Engineering, Stanford University, Stanford, California, USA.
Stephanie HarmanDepartment of Medicine, School of Medicine, Stanford University, Stanford, California, USA.
Heather ShawStanford Healthcare, Stanford, California, USA.
Ron LiDepartment of Medicine, School of Medicine, Stanford University, Stanford, California, USA.
Margaret SmithDepartment of Medicine, School of Medicine, Stanford University, Stanford, California, USA.
Kenny ShumDepartment of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA.
Jacob JavitzDepartment of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA.
Yohan VettethDepartment of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA.
Tina SetoDepartment of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA.
Steven C BagleyStanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA.ORCID 0000-0003-2187-7955
Nigam H ShahStanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA.ORCID 0000-0001-9385-7158
Stanford University · USStanford Medicine · USStanford Health Care · US

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

objectiveTo analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. MATERIALS AND

methodsWe built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP.

resultsWork capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care. DISCUSSION: The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit.

conclusionAn analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

Indexed as

Advance Care PlanningDelivery of Health CareElectronic Health RecordsHumansOutpatientsWorkflowlearning, evaluation, utility assessment, workflow simulation, advanced care planningmachine

Identifiers

PMID33355350
PMCPMC8200271
OpenAlexW3117058671

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.