Machine Learning to Identify Point-of-Care Ultrasound and Evaluate Standardized Documentation: Retrospective Operational Cohort Study.

Background: Point-of-care ultrasound (POCUS) is integral to obstetrics and gynecology (OBGYN), offering bedside diagnostic and therapeutic advantages. Despite its widespread adoption, accurate documentation and billing remain challenging due to inconsistent workflows, variable free-text note quality, and inefficiencies within electronic health record (EHR) systems. These barriers often result in missed procedural charges and hinder operational, educational, and reimbursement efforts. Objective: This study leveraged machine learning (ML) to automatically identify POCUS procedures within clinical notes and assessed the effect of implementing standardized procedure documentation (ProcDoc) templates on billing capture accuracy and efficiency. Methods: We conducted a multipart retrospective cohort study at a large academic medical center using EHRs from January 2018 to August 2024 across 11 OBGYN clinic sites. ML models (LightGBM [light gradient boosting machine] and BioClinBERT [biomedical and clinical bidirectional encoder representations from transformers]) were trained on clinical encounter notes to classify POCUS procedures and validated against Current Procedural Terminology (CPT) manual code assignments. In February 2023, a standardized ProcDoc smart form was introduced to streamline POCUS documentation and automatically trigger CPT billing codes. Preintervention and postintervention periods were compared using ML metrics and manual billing audits. Outcomes included model accuracy, recall, precision, adoption rates, improvement in billing recapture, and usage of ProcDoc templates. Results: A total of 559,029 encounters from 109,776 unique patients were analyzed. The BioClinBERT model (accuracy 0.97; F1-score 0.55-0.63) demonstrated a robust ability to identify documented and missed procedures in free-text clinical notes. ProcDoc adoption reached 75.1% within 12 months, supported by comprehensive staff education. Billing recapture-the proportion of charges missed by providers but later identified-dropped from 10.0% preintervention to 2.4% postintervention, primarily arising from the shift toward auto-capturing documentation (odds ratio 0.22, 95% CI 0.17-0.30; P<.001), with overall POCUS billing slightly increased (+0.6%). Most postintervention CPT codes (1812/2404, 75.4%) originated from ProcDoc templates, confirming improved workflow efficiency and reduced manual audit burden. Model analysis and billing metrics demonstrated that improvements were associated with workflow changes and not an increase in procedure frequency. Conclusions: ML modeling proved effective for extracting POCUS procedures from clinical documentation and serving as an evaluation tool for workflow interventions. Standardized documentation with ProcDoc significantly enhanced charge capture accuracy and reduced dependence on manual chart reviews and billing reconciliation. This approach highlights the use of ML as a retrospective auditing and evaluation tool for assessing clinical workflow interventions. Broader application of similar strategies could address documentation inefficiencies and promote sustainability across health care settings.