Automated survey collection with LLM-based conversational agents.

Objectives: Phone surveys are crucial for collecting health data but are expensive, time-consuming, and difficult to scale. To overcome these limitations, we propose a survey collection approach powered by conversational Large Language Models (LLMs). Materials and Methods: Our framework leverages an LLM-powered conversational agent to conduct surveys and transcribe conversations, along with an LLM (GPT-4o) to extract responses from the transcripts. We evaluated the framework's performance by analyzing transcription errors, the accuracy of inferred survey responses, and participant experiences across 40 survey responses collected from a convenience sample of 8 individuals, each adopting the role of five LLM-generated personas. Results: GPT-4o extracted responses to survey questions with an average accuracy of 98%, despite an average transcription word error rate of 7.7%. Participants reported occasional errors by the conversational agent but praised its ability to demonstrate comprehension and maintain engaging conversations. Discussion and Conclusion: Our study showcases the potential of LLM agents to enable scalable, AI-powered phone surveys, reducing human effort and advancing healthcare data collection.