Selecting, Scaling, and Measuring the Value of Ambient AI in a Nonacademic Health System: Multiphase Pilot Study.

Background: Most US health systems operate on a local or regional scale and face substantial financial and staffing pressures, which are intensified by challenges related to physician satisfaction and difficulties in recruitment and retention. Ambient artificial intelligence (AI) documentation solutions have the potential to reduce burdens and improve satisfaction, but vendor selection is often undermined by cognitive biases, unvalidated marketing claims, and limited real-world testing. Objective: To address this, McLeod Health developed and implemented an objective, multiphase approach to evaluate and adopt an ambient AI solution across its multihospital system. Methods: Our evaluation process began in spring 2024 with 4 leading vendors tested through live clinical simulations using 15 complex outpatient scripts, with organizational leaders serving as standardized patients. Ambient patient encounters were captured in real time, and AI-generated notes were scored by physicians, revenue cycle experts, and nonclinical reviewers for accuracy, billing quality, and readability. The top 2 vendors advanced to demonstrations of Epic workflow integration, with physician usability feedback guiding the final selection. In the third phase, the chosen vendor underwent a 90-day pilot across 5 ambulatory specialties beginning in October 2024, followed by system-wide implementation in January 2025. Key performance indicators included documentation time, coding, and financial trends, as well as patient and provider satisfaction. All statistical comparisons were 2-sided using a 95% CI. Results: The 3-phase evaluation process resulted in careful vendor selection. The pilot showed a 35.4% decrease in pajama time (P=.054, trend toward significance) and a 28.3% decrease in time in notes (n=23; P<.001). Coding patterns shifted toward higher-complexity visits, with a 3.8% increase in level 4 established patient visits (P=.05), and established patient volumes increased by 8.5%, which was associated with a projected revenue gain of US $2629 per provider per month. Patient satisfaction improved significantly across multiple domains, with large gains in listening, trust, communication, and treatment information (all P<.001). These gains exceeded those of prior system-wide patient satisfaction initiatives. System-wide rollout has achieved 81% adoption, with more than 150,000 notes generated. Conclusions: Our structured, multiphase evaluation process minimized vendor influence and cognitive bias during selection, validated results through real-world clinical testing, and enabled a system-wide rollout. This approach offers a practical framework for nonacademic health systems to objectively assess, implement, and scale ambient AI solutions while preserving fairness, transparency, and measurable value.