Evidence map›Paper›PMID 38929789›Full record

ArticleJournal of personalized medicine2024

Evaluation of ChatGPT as a Counselling Tool for Italian-Speaking MASLD Patients: Assessment of Accuracy, Completeness and Comprehensibility.

Nicola Pugliese, Davide Polverini, Rosa Lombardi, Grazia Pennisi, Federico Ravaioli, Angelo Armandi, Elena Buzzetti, Andrea Dalbeni, Antonio Liguori, Alessandro Mantovani and 9 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Journal of personalized medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 11 papers.

0numbers the graph read from it
0cells of the map it votes in
11citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

11 citing papers in PubMed.

  1. Review
  2. Article
  3. Review
  4. Review
  5. Review
  6. Article
  7. Review
  8. Article
  9. Review
  10. Article
  11. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

19 authors.

Nicola PuglieseDepartment of Biomedical Sciences, Humanitas University, Pieve Emanuele, 20072 Milan, Italy.ORCID 0000-0001-6466-1412
Davide PolveriniDepartment of Biomedical Sciences, Humanitas University, Pieve Emanuele, 20072 Milan, Italy.
Rosa LombardiUnit of Internal Medicine and Metabolic Disease, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico of Milan, 20122 Milan, Italy.ORCID 0000-0001-6958-927X
Grazia PennisiSection of Gastroenterology and Hepatology, PROMISE, University of Palermo, 90127 Palermo, Italy.
Federico RavaioliDepartment of Medical and Surgical Sciences (DIMEC), University of Bologna, 40138 Bologna, Italy.ORCID 0000-0002-1142-8585
Angelo ArmandiDivision of Gastroenterology and Hepatology, Department of Medical Sciences, University of Turin, Corso Dogliotti 14, 10126 Turin, Italy.ORCID 0000-0002-7245-4445
Elena BuzzettiInternal Medicine and Centre for Hemochromatosis and Hereditary Liver Diseases, ERN-EuroBloodNet Center for Iron Disorders, Azienda Ospedaliero-Universitaria di Modena-Policlinico, 41125 Modena, Italy.ORCID 0000-0002-4462-7935
Andrea DalbeniDivision of General Medicine C, Department of Medicine, University and Azienda Ospedaliera Universitaria Integrata of Verona, University of Verona, 37134 Verona, Italy.ORCID 0000-0003-0402-4176
Antonio LiguoriDiSMeC-Department of Scienze Mediche e Chirurgiche, Fondazione Policlinico Gemelli IRCCS, 00168 Rome, Italy.ORCID 0000-0002-0801-7152
Alessandro MantovaniSection of Endocrinology, Diabetes and Metabolism, Department of Medicine, University and Azienda Ospedaliera Universitaria Integrata of Verona, Piazzale Stefani, 37126 Verona, Italy.ORCID 0000-0002-7271-6329
Rosanna VillaniC.U.R.E. (University Center for Liver Disease Research and Treatment), Liver Unit, Department of Medical and Surgical Sciences, University of Foggia, 71122 Foggia, Italy.ORCID 0000-0001-9875-019X
Ivan GardiniEpaC Onlus, Italian Liver Patient Association, 10141 Turin, Italy.
Cesare HassanDepartment of Biomedical Sciences, Humanitas University, Pieve Emanuele, 20072 Milan, Italy.
Luca ValentiDepartment of Pathophysiology and Transplantation, Università degli Studi di Milano, 20122 Milan, Italy.
Luca MieleDiSMeC-Department of Scienze Mediche e Chirurgiche, Fondazione Policlinico Gemelli IRCCS, 00168 Rome, Italy.ORCID 0000-0003-3464-0068
Salvatore PettaSection of Gastroenterology and Hepatology, PROMISE, University of Palermo, 90127 Palermo, Italy.
Giada SebastianiDivision of Gastroenterology and Hepatology, McGill University Health Centre, Montreal, QC H4A 3J1, Canada.ORCID 0000-0003-2655-8283
Alessio AghemoDepartment of Biomedical Sciences, Humanitas University, Pieve Emanuele, 20072 Milan, Italy.
Nafld Expert Chatbot Working Group

Funding

Fonds de Recherche du Quebec - Sante (FRQS) 296306Italian PNRR-MAD-2022-12375656 project, PRIN 2022 2022L273C9 project and RF-2021-12372399 project
6 · The paper itself

Abstract

backgroundArtificial intelligence (AI)-based chatbots have shown promise in providing counseling to patients with metabolic dysfunction-associated steatotic liver disease (MASLD). While ChatGPT3.5 has demonstrated the ability to comprehensively answer MASLD-related questions in English, its accuracy remains suboptimal. Whether language influences these results is unclear. This study aims to assess ChatGPT's performance as a counseling tool for Italian MASLD patients.

methodsThirteen Italian experts rated the accuracy, completeness and comprehensibility of ChatGPT3.5 in answering 15 MASLD-related questions in Italian using a six-point accuracy, three-point completeness and three-point comprehensibility Likert's scale.

resultsMean scores for accuracy, completeness and comprehensibility were 4.57 ± 0.42, 2.14 ± 0.31 and 2.91 ± 0.07, respectively. The physical activity domain achieved the highest mean scores for accuracy and completeness, whereas the specialist referral domain achieved the lowest. Overall, Fleiss's coefficient of concordance for accuracy, completeness and comprehensibility across all 15 questions was 0.016, 0.075 and -0.010, respectively. Age and academic role of the evaluators did not influence the scores. The results were not significantly different from our previous study focusing on English.

conclusionLanguage does not appear to affect ChatGPT's ability to provide comprehensible and complete counseling to MASLD patients, but accuracy remains suboptimal in certain domains.

Indexed as

artificial intelligencechatbotcounsellingdietMASLDphysical activitysteatosis

Identifiers

PMID38929789
PMCPMC11204905

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.