Can LLMs effectively assist medical coding? Evaluating GPT performance on DRG and targeted clinical tasks.

backgroundLarge language Models (LLM) have demonstrated near human-level performance in medical domain, from summarizing clinical notes and passing medical licensing examinations, to predictive tasks such as disease diagnoses and treatment recommendations. However, currently there is little research on their efficacy for medical coding, a pivotal component in health informatics, clinical trials, and reimbursement management. This study proposes a prompt framework and investigates its effectiveness for medical coding.

methodsFirst, a prompt-based learning framework is proposed. This framework aims to improve the performance of GPT for complex medical coding tasks by augmenting it with specific prior knowledge, utilizing techniques such as meta prompt, many-shot learning, and dynamic in-context learning. Then its effectiveness is evaluated with three different medical coding tasks. Finally, ablation studies are presented to validate and analyze the contribution of each module in the proposed prompt framework.

resultsFor the MIMIC-IV dataset, the prediction accuracies of top-1 and top-5 for the 30 most frequent MS-DRG codes are 68.1% and 90.0%, respectively. The clinical trial criteria coding task results in a macro-F1 score of 68.4 on the CHIP-CTC test dataset in Chinese. Both results are comparable to the best methods in comparison that training deep leaning models or fine-tuning LLMs.

conclusionsThis study demonstrates that for targeted coding tasks, DRG and other simplified scenarios, off-the-shelf GPT models guided by carefully designed prompts can achieve performance comparable to state-of-the-art methods. While current GPT models serve as helpful assistants to human coding specialists,they are not yet equipped to fully replace expert judgment. Importantly, the recent and drastic decline of the cost of deploying large language models makes their integration into medical coding workflows increasingly feasible. As these models continue to evolve, their potential to provide reliable support for DRG coding task is likely to grow rapidly over time.

trial registrationNot applicable.