Evidence map›Paper›PMID 35013569›Full record

ArticleNPJ digital medicine2022

Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review.

Anne A H de Hond, Artuur M Leeuwenberg, Lotty Hooft, Ilse M J Kant, Steven W J Nijman, Hendrikus J A van Os, Jiska J Aardoom, Thomas P A Debray, Ewoud Schuit, Maarten van Smeden and 4 more

Abstract readScoping Review
In one paragraph

Article in NPJ digital medicine, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 208 papers, 20 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
208citing papers in PubMed, 20 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

208 citing papers in PubMed, 20 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Guideline
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Pooled it
  16. Pooled it
  17. Pooled it
  18. Pooled it
  19. Pooled it
  20. Pooled it

148 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

14 authors.

Anne A H de Hond *Department of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands. a.a.h.de_hond@lumc.nl.
Artuur M Leeuwenberg *Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands. a.m.leeuwenberg-15@umcutrecht.nl.ORCID http://orcid.org/0000-0002-2892-0285
Lotty HooftJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
Ilse M J KantDepartment of Information Technology and Digital Innovation, Leiden University Medical Center, Leiden, The Netherlands.
Steven W J NijmanJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.ORCID http://orcid.org/0000-0001-6798-2078
Hendrikus J A van OsClinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands.
Jiska J AardoomNational eHealth Living Lab, Leiden, The Netherlands.ORCID http://orcid.org/0000-0002-4023-6814
Thomas P A DebrayJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.ORCID http://orcid.org/0000-0002-1790-2719
Ewoud SchuitJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.ORCID http://orcid.org/0000-0002-9548-3214
Maarten van SmedenJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
Johannes B ReitsmaJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
Ewout W SteyerbergClinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands.
Niels H ChavannesNational eHealth Living Lab, Leiden, The Netherlands.
Karel G M MoonsJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.

Identifiers

PMID35013569
PMCPMC8748878

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.