Evidence map›Paper›PMID 31022752›Full record

ReviewYearbook of medical informatics2019

Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications.

Farah Magrabi, Elske Ammenwerth, Jytte Brender McNair, Nicolet F De Keizer, Hannele Hyppönen, Pirkko Nykänen, Michael Rigby, Philip J Scott, Tuulikki Vehko, Zoie Shui-Yee Wong and 1 more

Abstract readReview
PubMed Full text at PMC Publisher
In one paragraph

Review in Yearbook of medical informatics, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 124 papers, 7 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
124citing papers in PubMed, 7 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

124 citing papers in PubMed, 7 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Guideline
  8. Article
  9. Article
  10. What do clinicians edit in ambient AI-drafted clinical documentation? A qualitative content analysis.Journal of the American Medical Informatics Association : JAMIA · 2026
    Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
  16. Article
  17. Article
  18. Article
  19. Article
  20. Article

64 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

11 authors.

Farah MagrabiMacquarie University, Australian Institute of Health Innovation, Sydney, Australia.
Elske AmmenwerthUMIT, University for Health Sciences, Medical Informatics and Technology, Institute of Medical Informatics, Hall in Tyrol, Austria.
Jytte Brender McNairDepartment of Health Science and Technology, Aalborg University, Aalborg, Denmark.
Nicolet F De KeizerAmsterdam UMC, University of Amsterdam, Department of Medical Informatics, Amsterdam Public Health research institute, The Netherlands.
Hannele HyppönenNational Institute for Health and Welfare, Information Department, Helsinki, Finland.
Pirkko NykänenTampere University, Faculty for Information Technology and Communication Sciences, Tampere, Finland.
Michael RigbyKeele University, School of Social Science and Public Policy, Keele, United Kingdom.
Philip J ScottUniversity of Portsmouth, Centre for Healthcare Modelling and Informatics, Portsmouth, United Kingdom.
Tuulikki VehkoNational Institute for Health and Welfare, Information Department, Helsinki, Finland.
Zoie Shui-Yee WongSt. Luke's International University, Tokyo, Japan.
Andrew GeorgiouMacquarie University, Australian Institute of Health Innovation, Sydney, Australia.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

objectivesThis paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance.

methodA narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems.

resultsThere is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed.

conclusionCommitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

Indexed as

Artificial IntelligenceDecision Support Systems, ClinicalEvaluation Studies as TopicMachine LearningProgram Evaluation

Identifiers

PMID31022752
PMCPMC6697499

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC-ND
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.