When scientific constraints and real-world implementation collide: Lessons from a hybrid type 3 study of a digital, direct-to-consumer HIV prevention program.

Background: Digital HIV prevention interventions (DHIs) are efficacious in increasing prevention behaviors, and delivering them direct-to-consumer (DTC) expands their reach. However, few have successfully moved rigorously-studied DHIs from research to public health practice. Hybrid type 3 effectiveness-implementation studies promise to approximate more naturalistic settings and accelerate translation from research to practice, and few DTC DHIs have been tested this way. This study describes lessons learned from a hybrid type 3 study of a DTC DHI for young men who have sex with men (YMSM) ages 18-29 called Keep It Up! (KIU!) 3.0. Methods: KIU! 3.0 was initially designed as a cluster-randomized hybrid type 3 implementation-effectiveness study across 44 United States counties with high HIV incidence among YMSM (22 DTC; 22 implemented in community-based organizations). The DTC strategy relied on online recruitment, at-home HIV/STI testing, and centralized intervention delivery. Over the course of the trial (October 2019-March 2023), we adapted our implementation four times in response to recruitment and retention challenges. Data sources included enrollment logs, advertising expenditures, participant communications, and internal documentation, which are used to characterize recruitment, costs, and recruitment/retention patterns. Results: Due to challenges in recruitment and retention, four major changes were made over the course of the trial: 1) streamlining enrollment procedures, including shortening screening and verification steps, 2) modifying eligibility criteria, including expanding age limits, removing sexual risk requirements, and including gender-diverse participants, 3) introducing and increasing financial incentives for intervention completing and follow-up measures, and 4) shifting from county-level to nationwide recruitment. Following this fourth change in March 2021, enrollment increased by 301%, and cost per enrolled participant decreased. Ultimately 1,468 participants were enrolled nationwide. Nevertheless, retention and return of at-home STI test kits remained challenging, and guaranteed incentives increased the prevalence of imposter participants. Conclusion: Implementing a DTC DHI within a hybrid type 3 study required balancing pragmatic implementation goals with effectiveness outcomes measurement. Future hybrid studies of DTC digital health interventions should consider eligibility criteria, incentive structures, outcomes measurement strategies, and the distinction between research and service components to better align research with real-world implementation contexts.