Mapping ICU nursing intervention notes to clinical care classification using a large language model: an evaluation study.

backgroundIntensive care unit (ICU) nurses often document their clinical concerns as free-text formats. The free-text documents pose significant challenges for transformation into quantifiable and standardized nursing data. Although large language models (LLMs) show promises in standardizing such documents, evidence supporting their application in nursing domain-specific tasks remains limited. This study aims to evaluate the extent to which an LLM can map ICU nursing intervention notes to Clinical Care Classification (CCC) terminologies and to assess the quality of the mapping results.

methodsThis evaluation study utilized the Medical Information Mart for Intensive Care (MIMIC) IV database. Nursing notes documented during ICU stays were included, and those without records of interventions or actions were excluded. These selected notes were defined as intervention notes and were analyzed using a four-stage process: (1) expert-driven development of CCC mapping references; (2) LLM-driven generation of CCC mappings; (3) quantitative, and (4) qualitative evaluation of LLM-driven CCC mappings against expert-driven references. The LLM-driven mapping was conducted using prompt engineering techniques and employed the GPT-4o mini model deployed on Azure OpenAI to ensure data security.

resultsA total of 9,614,214 intervention note entries from 39,711 ICU admissions were extracted from MIMIC-IV. Among these, duplicates were excluded, and 269 unique intervention notes were selected. Of these 269 intervention notes, 107 were identified as reflecting nurses' concerns. Based on these 107 intervention notes, CCC mapping references were established by ICU nurses and subject matter experts, comprising 17 care components, 31 interventions, and 26 sub-interventions. Approximately two-thirds of the LLM-driven CCC mappings achieved a performance comparable to those of expert-driven mappings. Quantitative and qualitative evaluations showed that the LLM captured underlying meaning embedded in nursing intervention notes relatively well, with inconsistency due to reasoning failure as the primary source of mapping errors.

conclusionsThis study identified the feasibility of LLM-driven automated nursing terminology mapping from ICU nursing intervention notes to CCC terms, potentially reducing the burden of terminology mapping. Future research should aim to improve the model's contextual understanding by utilizing larger, multiple, narrative data-rich datasets, thereby enhancing its generalizability. CLINICAL TRIAL NUMBER: Not applicable.