Evidence map›Paper›PMID 38070173›Full record

ArticleJournal of the American Medical Informatics Association : JAMIA2024

Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review.

Hang Ding, Joshua Simmich, Atiyeh Vaezipour, Nicole Andrews, Trevor Russell

Abstract readScoping Review
PubMed Full text at PMC Publisher
In one paragraph

Article in Journal of the American Medical Informatics Association : JAMIA, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers, 1 of them a synthesis that pooled it.

0numbers the graph read from it
0cells of the map it votes in
22citing papers in PubMed, 1 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

22 citing papers in PubMed, 1 synthesis or guideline pooled it.

  1. Pooled it
  2. Trial
  3. AI-Augmented Prevention Science Needs Community-Engaged Prevention Science: a Framework for Greater Accountability.Prevention science : the official journal of the Society for Prevention Research · 2026
    Article
  4. Article
  5. Article
  6. Article
  7. Review
  8. Review
  9. Article
  10. Article
  11. Article
  12. Article
  13. Review
  14. Article
  15. Article
  16. Review
  17. Article
  18. Large language models and generative AI in telehealth: a responsible use lens.Journal of the American Medical Informatics Association : JAMIA · 2024
    Review
  19. Celebrating Eta Berner and her influence on biomedical and health informatics.Journal of the American Medical Informatics Association : JAMIA · 2024
    Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

5 authors.

Hang DingRECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia.
Joshua SimmichRECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia.
Atiyeh VaezipourRECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia.
Nicole AndrewsRECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia.
Trevor RussellRECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia.

Funding

University of Queensland
6 · The paper itself

Abstract

objectivesConversational agents (CAs) with emerging artificial intelligence present new opportunities to assist in health interventions but are difficult to evaluate, deterring their applications in the real world. We aimed to synthesize existing evidence and knowledge and outline an evaluation framework for CA interventions. MATERIALS AND

methodsWe conducted a systematic scoping review to investigate designs and outcome measures used in the studies that evaluated CAs for health interventions. We then nested the results into an overarching digital health framework proposed by the World Health Organization (WHO).

resultsThe review included 81 studies evaluating CAs in experimental (n = 59), observational (n = 15) trials, and other research designs (n = 7). Most studies (n = 72, 89%) were published in the past 5 years. The proposed CA-evaluation framework includes 4 evaluation stages: (1) feasibility/usability, (2) efficacy, (3) effectiveness, and (4) implementation, aligning with WHO's stepwise evaluation strategy. Across these stages, this article presents the essential evidence of different study designs (n = 8), sample sizes, and main evaluation categories (n = 7) with subcategories (n = 40). The main evaluation categories included (1) functionality, (2) safety and information quality, (3) user experience, (4) clinical and health outcomes, (5) costs and cost benefits, (6) usage, adherence, and uptake, and (7) user characteristics for implementation research. Furthermore, the framework highlighted the essential evaluation areas (potential primary outcomes) and gaps across the evaluation stages. DISCUSSION AND

conclusionThis review presents a new framework with practical design details to support the evaluation of CA interventions in healthcare research. PROTOCOL REGISTRATION: The Open Science Framework (https://osf.io/9hq2v) on March 22, 2021.

Indexed as

Artificial IntelligenceCommunicationEvaluation Studies as TopicHumanschatbotconversational agentevaluationhealthcaresystematic reviewvirtual assistant

Identifiers

PMID38070173
PMCPMC10873847

What OpenQuestion holds

Textmetadata
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.