Evidence map›Paper›PMID 38914707›Full record

Observational studyScientific reports2024

Constructing and implementing a performance evaluation indicator set for artificial intelligence decision support systems in pediatric outpatient clinics: an observational study.

Yingwen Wang, Weijia Fu, Yuejie Zhang, Daoyang Wang, Ying Gu, Weibing Wang, Hong Xu, Xiaoling Ge, Chengjie Ye, Jinwu Fang and 5 more

Abstract readObservational Study
In one paragraph

Observational study in Scientific reports, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 2 papers.

0numbers the graph read from it
0cells of the map it votes in
2citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

2 citing papers in PubMed.

  1. Article
  2. Pediatrics 4.0: the Transformative Impacts of the Latest Industrial Revolution on Pediatrics.Health care analysis : HCA : journal of health philosophy and policy · 2025
    Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

15 authors.

Yingwen Wang *Nursing Department, Children's Hospital of Fudan University, Shanghai, 201102, China.
Weijia Fu *Medical Information Center, Children's Hospital of Fudan University, Shanghai, 201102, China.
Yuejie ZhangSchool of Computer Science, Fudan University, Shanghai, 200438, China.
Daoyang WangSchool of Public, Health Fudan University, Shanghai, 200032, China.
Ying GuNursing Department, Children's Hospital of Fudan University, Shanghai, 201102, China.
Weibing WangSchool of Public, Health Fudan University, Shanghai, 200032, China.
Hong XuNephrology Department, Children's Hospital of Fudan University, Shanghai, 201102, China.
Xiaoling GeStatistical and Data Management Center, Children's Hospital of Fudan University, Shanghai, 201102, China.
Chengjie YeMedical Information Center, Children's Hospital of Fudan University, Shanghai, 201102, China.
Jinwu FangSchool of Public, Health Fudan University, Shanghai, 200032, China.
Ling SuStatistical and Data Management Center, Children's Hospital of Fudan University, Shanghai, 201102, China.
Jiayu WangNational Health Commission Key Laboratory of Neonatal Diseases (Fudan University), Children's Hospital of Fudan University, Shanghai, 201102, China.
Wen HeRespiratory Department, Children's Hospital of Fudan University, Shanghai, 201102, China.
Xiaobo ZhangRespiratory Department, Children's Hospital of Fudan University, Shanghai, 201102, China. zhangxiaobo0307@163.com.
Rui FengSchool of Computer Science, Fudan University, Shanghai, 200438, China. fengrui@fudan.edu.cn.

Funding

2021 Artificial Intelligence Technology Support Special Directional Project (The second batch) 21002411800National Key R&D Program of China 2021ZD0113504Science and Technology Commission of Shanghai Municipality 21511104502Shanghai Municipal Hospital Pediatric Specialist Alliance SHDC22021305-A, SHDC22021305-B
6 · The paper itself

Abstract

Artificial intelligence (AI) decision support systems in pediatric healthcare have a complex application background. As an AI decision support system (AI-DSS) can be costly, once applied, it is crucial to focus on its performance, interpret its success, and then monitor and update it to ensure ongoing success consistently. Therefore, a set of evaluation indicators was explicitly developed for AI-DSS in pediatric healthcare, enabling continuous and systematic performance monitoring. The study unfolded in two stages. The first stage encompassed establishing the evaluation indicator set through a literature review, a focus group interview, and expert consultation using the Delphi method. In the second stage, weight analysis was conducted. Subjective weights were calculated based on expert opinions through analytic hierarchy process, while objective weights were determined using the entropy weight method. Subsequently, subject and object weights were synthesized to form the combined weight. In the two rounds of expert consultation, the authority coefficients were 0.834 and 0.846, Kendall's coordination coefficient was 0.135 in Round 1 and 0.312 in Round 2. The final evaluation indicator set has three first-class indicators, fifteen second-class indicators, and forty-seven third-class indicators. Indicator I-1(Organizational performance) carries the highest weight, followed by Indicator I-2(Societal performance) and Indicator I-3(User experience performance) in the objective and combined weights. Conversely, 'Societal performance' holds the most weight among the subjective weights, followed by 'Organizational performance' and 'User experience performance'. In this study, a comprehensive and specialized set of evaluation indicators for the AI-DSS in the pediatric outpatient clinic was established, and then implemented. Continuous evaluation still requires long-term data collection to optimize the weight proportions of the established indicators.

Indexed as

Artificial IntelligenceDecision Support Systems, ClinicalAmbulatory Care FacilitiesChildHumansPediatrics

Identifiers

PMID38914707
PMCPMC11196575

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.