Evidence map›Paper›PMID 36120717›Full record

ArticleFrontiers in digital health2022

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings.

Sharon E Davis, Colin G Walsh, Michael E Matheny

Abstract read
In one paragraph

Article in Frontiers in digital health, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers.

0numbers the graph read from it
0cells of the map it votes in
15citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

15 citing papers in PubMed.

  1. Review
  2. Article
  3. Review
  4. Article
  5. Article
  6. Review
  7. Article
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Sustainable deployment of clinical prediction tools-a 360° approach to model maintenance.Journal of the American Medical Informatics Association : JAMIA · 2024
    Article
  14. Pathophysiological Features in Electronic Medical Records Sustain Model Performance under Temporal Dataset Shift.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2024
    Article
  15. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

3 authors.

Sharon E DavisDepartment of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States.
Colin G WalshDepartment of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States.
Michael E MathenyDepartment of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States.

Funding

Overall: Eunice Kennedy Shriver Intellectual and Developmental Disabilities Research Center at VanderbiltP50HD103537 · NICHD · VANDERBILT UNIVERSITY MEDICAL CENTER · PI Jeffrey L Neul · 2020 to 2026
$10.3M
Distinguishing Clinical and Genetic Risk of Suicidal Ideation from Attempts to Inform PreventionR01MH121455 · NIMH · VANDERBILT UNIVERSITY MEDICAL CENTER · PI RUDERFER, DOUGLAS, WALSH, COLIN GEORGE · 2020 to 2024
$2.7M
Integrating signals of suicide risk from DoD and VHA data to improve upon suicide risk prevention strategies for combat VeteransR01MH120122 · NIMH · UNIVERSITY OF COLORADO DENVER · PI ADAMS, RACHEL SAYKO, BRENNER, LISA A · 2019 to 2022
$1.9M
2/2 Leveraging electronic health records for pharmacogenomics of psychiatric diordersR01MH116269 · NIMH · VANDERBILT UNIVERSITY MEDICAL CENTER · PI RUDERFER, DOUGLAS, WALSH, COLIN GEORGE · 2019 to 2022
$1.7M
NICHD NIH HHS P50 HD103537NIMH NIH HHS R01 MH116269NIMH NIH HHS R01 MH120122NIMH NIH HHS R01 MH121455
6 · The paper itself

Abstract

As the implementation of artificial intelligence (AI)-enabled tools is realized across diverse clinical environments, there is a growing understanding of the need for ongoing monitoring and updating of prediction models. Dataset shift-temporal changes in clinical practice, patient populations, and information systems-is now well-documented as a source of deteriorating model accuracy and a challenge to the sustainability of AI-enabled tools in clinical care. While best practices are well-established for training and validating new models, there has been limited work developing best practices for prospective validation and model maintenance. In this paper, we highlight the need for updating clinical prediction models and discuss open questions regarding this critical aspect of the AI modeling lifecycle in three focus areas: model maintenance policies, performance monitoring perspectives, and model updating strategies. With the increasing adoption of AI-enabled tools, the need for such best practices must be addressed and incorporated into new and existing implementations. This commentary aims to encourage conversation and motivate additional research across clinical and data science stakeholders.

Indexed as

artificial intelligencedataset shiftmachine learningmodel updatingrisk model surveillance

Identifiers

PMID36120717
PMCPMC9478183

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.