Evidence map›Paper›PMID 41708070›Full record

ArticleJournal of applied clinical medical physics2026

Leveraging large language models for heuristic usability assessment of medical software: Insights with the Radiation Planning Assistant.

Laurence E Court, Jacobus Smit, Lourens Strauss, William Shaw, Andrea Marais, Christoph Trauernicht, Nanette Joubert, Elaine Smith, Shona Badre, Graeme L Lazarus and 12 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Journal of applied clinical medical physics, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 1 paper.

0numbers the graph read from it
0cells of the map it votes in
1citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

1 citing paper in PubMed.

  1. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

22 authors.

Laurence E CourtDepartment of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.
Jacobus SmitDepartment of Medical Physics, University of the Free State School of Medicine, Bloemfontein, South Africa.
Lourens StraussDepartment of Medical Physics, University of the Free State School of Medicine, Bloemfontein, South Africa.ORCID https://orcid.org/0000-0002-2607-7941
William ShawDepartment of Medical Physics, University of the Free State School of Medicine, Bloemfontein, South Africa.
Andrea MaraisDivision of Medical Physics, Tygerberg Hospital, Stellenbosch University, Cape Town, South Africa.ORCID https://orcid.org/0000-0002-5040-5896
Christoph TrauernichtDivision of Medical Physics, Tygerberg Hospital, Stellenbosch University, Cape Town, South Africa.
Nanette JoubertDepartment of Radiation Medicine, Groote Schuur Hospital, University of Cape Town, Cape Town, South Africa.
Elaine SmithDepartment of Radiation Medicine, Groote Schuur Hospital, University of Cape Town, Cape Town, South Africa.
Shona BadreDepartment of Radiation Oncology, Inkosi Albert Luthuli Central Hospital, Durban, South Africa.
Graeme L LazarusDepartment of Radiation Oncology, Inkosi Albert Luthuli Central Hospital, Durban, South Africa.
Thekiso KhotleDepartment of Radiation Oncology, Charlotte Maxeke Johannesburg Academic Hospital, Wits University, Johannesburg, South Africa.ORCID https://orcid.org/0009-0003-5211-8406
Lauren NethertonHouston, Texas, USA.
Wanda van HeerdenIcon Oncology, Johannesburg, South Africa.
Carlos CardenasDepartment of Radiation Oncology, University of Alabama at Birmingham, Birmingham, Alabama, USA.
Monica SerbanDepartment of Radiation Oncology, Princess Margaret Cancer Centre, Toronto, Canada.
Jan SeuntjensDepartment of Radiation Oncology, Princess Margaret Cancer Centre, Toronto, Canada.
Christine V ChungDepartment of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.ORCID https://orcid.org/0000-0002-3093-6913
Pavel GovyadinovDepartment of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.ORCID https://orcid.org/0000-0001-7980-2791
Meena KhanDepartment of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.
Saurabh NairDepartment of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.
Tucker NethertonDepartment of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.ORCID https://orcid.org/0000-0003-1583-7121
Lifei ZhangDepartment of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

backgroundUsability engineering is essential for ensuring the safety and effectiveness of medical software, as design-related issues are a leading cause of use errors in clinical settings. Heuristic evaluation provides a practical approach to identifying usability problems, but its outcomes depend heavily on expert interpretation. Large Language Models (LLMs), such as ChatGPT, offer a potential means to augment heuristic evaluation by generating structured, context-aware usability feedback. This study explored the use of ChatGPT to support heuristic assessment of the Radiation Planning Assistant (RPA), a web-based radiotherapy planning tool designed to support clinical teams in low- and middle-income countries.

methodsChatGPT was provided with the RPA user and technical guides, training videos for each functional dashboard, and Zhang et al.'s 14 usability heuristics. The model was instructed to score each dashboard according to these heuristics, using Zhang's 0-4 severity scale, and to propose concrete interface improvements. The resulting feedback was reviewed and scored independently by the RPA developer team and by 13 users during a dedicated User Meeting. Comparative analysis was performed between ChatGPT, developer, and user ratings.

resultsChatGPT identified 26 potential usability issues across six heuristic domains. The developer team considered nine of these actionable, though all were classified as minor (severity ≤ 2). User ratings showed wide variability, with nine suggestions achieving mean scores ≥ 1.5. Qualitative agreement between users and developers was limited, underscoring the importance of diverse perspectives in heuristic evaluation. Three suggestions-enhanced upload logs, reversible actions ("reopen request"), and stronger error prevention-were rated as potentially high priority by a minority of users. ChatGPT's ratings were consistent across dashboards.

conclusionsWhile ChatGPT did not reveal any critical usability failures, its heuristic assessment proved valuable in prompting discussion, identifying minor refinements, and enriching both developer and user engagement with the RPA's interface design. This study demonstrates that LLMs can serve as an effective, low-cost complement to conventional heuristic evaluation, supporting early-stage usability review and stakeholder dialogue in the development of medical software.

Indexed as

HeuristicsLarge Language ModelsRadiotherapy Planning, Computer-AssistedSoftwareUser-Computer InterfaceHumansHeuristic evaluationLarge language modelsUsabilityUser interface design

Identifiers

PMID41708070
PMCPMC12916175

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.