Evaluating Methodological Coherence and Evidence Recognition in Digital Health Systematic Reviews: Sample-based Meta-research Study.

Background: Despite a growing number of systematic reviews on digital health interventions, many do not sufficiently support the recognition of conclusive evidence. Methodological shortcomings may impede the identification and communication of robust findings. Abstracts are the basis for study selection in systematic reviews and are increasingly used in automated screening processes and rapid assessments. Objective: This meta-research study examines to what extent systematic reviews apply methodological standards-particularly the specification of PICO (population or problem, intervention, comparison, and outcome) elements-and how this relates to the likelihood of conclusive evidence recognition. It is based on a random sample and focuses on the assessment at the abstract level, as abstracts are used independently of the specific review choice to screen and select studies for evidence synthesis, making them critical for evidence recognition. Methods: Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we conducted a comprehensive database search (2011-2023). From 2528 eligible systematic reviews, a random sample of 250 abstracts was analyzed descriptively. Abstracts were assessed for PICO specification and evidence conclusiveness in the context of further study characteristics. Results: In total, 48% (119/250) of reviews showed low or very low PICO specification, and 64% (159/250) reported inconclusive or weak evidence. Higher specification of outcomes and problems was moderately associated with conclusive evidence. Beside the formulation of the research question along the PICO scheme, we identified recurring issues in search and screening strategy design (eg, limited database use, vague search terms, and long search periods), restrictive eligibility criteria (eg, exclusive reliance on randomized controlled trials), inconsistent use of quality appraisal tools, and underusage of alternative synthesis methods to hinder evidence recognition. Conclusions: Our findings suggest that methodological coherence across all review stages is a necessary condition to ensure conclusions and evidence-informed decisions in the digitalization of health care are both valid and meaningful. A structured PICO-based framework, which is aligned with current research, builds on well-established categories and provides clear and differentiated definitions that may enhance the focus and evidentiary strength of future reviews.