A comprehensive qualitative analysis of patient dialogue summarization using large language models applied to noisy, informal, non-English real-world data.

This study evaluates the ability of Large Language Models (LLMs) to summarize real-world dialogues between patients and the healthcare team of an e-health company that provides digital healthcare services, primarily communicating via WhatsApp. The team needs quick access to patient information to deliver accurate and personalized responses. Summarizing past messages is the approach examined here, aiming for concise, non-redundant, and truthful summaries that capture the main dialogue characteristics despite facing (real-world) noisy and informal content in an under-represented language - Portuguese. To do so, we collected an anonymized Portuguese dataset of WhatsApp messages exchanged between patients and the healthcare team. Dialogue quality was assessed for size, readability, and correctness before generating summaries with LLaMA3 and Qwen2 using specific prompts. Volunteers evaluated these summaries on coverage, relevance, redundancy, and veracity using a 5-point Likert scale. Our qualitative and quantitative experimental results indicate that LLMs can produce effective summaries of dialogues between patients and healthcare teams, even when faced with low-quality data in an underrepresented language. This is a surprising result due to the challenging scenario. Among the tested LLMs, LLaMA3 demonstrated a slight edge over QWen2 in coverage and veracity among the evaluated methods. Our results demonstrate a potential to build real-world practical services to assist healthcare professionals in responding to patient messages with agility, clarity, and cohesion, enhancing both communication efficiency and patient satisfaction. Ultimately, the advocated approach could significantly improve the landscape of online healthcare communication, particularly in resource-constrained settings like Brazil, where access to primary care is limited.