SynthesisJournal of the American Medical Informatics Association : JAMIA2021
Toolkits for implementing and evaluating digital health: A systematic review of rigor and reporting.
Synthesis in Journal of the American Medical Informatics Association : JAMIA, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 20 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
20 citing papers in PubMed, 1 synthesis or guideline pooled it.
- The State of the Art of Telemedicine Implementation Architecture: Rapid Umbrella Review of Systematic Reviews.Journal of medical Internet research · 2025Pooled it
- Validating DigiHEALTHQUAL for measuring digital health effects on service quality in outpatient care in Germany.NPJ digital medicine · 2026Article
- Developing a Toolkit to Support Dissemination and Implementation of the Red Carpet Entry Program.AIDS and behavior · 2026Article
- Toolkits and Toolboxes on Cancer Care and Mental Health: A Scoping Review Following the PRSMA-ScR Guidelines.Healthcare (Basel, Switzerland) · 2026Review
- Co-Designing and Evaluating a Digital Competencies Toolkit for Nursing Students.Journal of advanced nursing · 2026Article
- Frameworks for assessing digital health technologies: a scoping review.International journal of technology assessment in health care · 2026Article
- Recurring implementation determinants in digital health innovations: a multi-context multiple case study and cross-case synthesis into a four-domain analytical framework.Frontiers in digital health · 2026Article
- Digital health maturity models for Human Resources for Health Information Systems in low- and middle-income countries in Africa: A scoping review.Journal of public health in Africa · 2026Review
- Quality improvement project to transition psychosocial oncology clinical care to a telehealth workflow during the COVID-19 pandemic: a quasi-experimental study.BMC health services research · 2025Article
- Service Quality Assessment of Digital Health Solutions in Outpatient Care: Qualitative Item Repository Development Study.JMIR formative research · 2025Article
- Digital interventions as part of routine addiction care in Sweden: healthcare staff perceptions of what works.Addiction science & clinical practice · 2025Article
- Understanding the user experience of a mobile produce market intervention toolkit.Translational behavioral medicine · 2025Article
- Primary Care EHR data on Social Determinants of Health: Quality and Fitness for Purpose in Precision/Personalised Medicine.Yearbook of medical informatics · 2024Article
- Semantic Interoperability of Electronic Health Records: Systematic Review of Alternative Approaches for Enhancing Patient Information Availability.JMIR medical informatics · 2024Review
- Defining and Scoping Participatory Health Informatics: An eDelphi Study.Methods of information in medicine · 2023Article
- Translation, cultural adaptation and validity assessment of the Dutch version of the eHealth Literacy Questionnaire: a mixed-method approach.BMC public health · 2023Article
- Digital health and capability maturity models-a critical thematic review and conceptual synthesis of the literature.Journal of the American Medical Informatics Association : JAMIA · 2023Review
- "Digital Health Diplomacy" in Global Digital Health? A call for critique and discourse.Journal of the American Medical Informatics Association : JAMIA · 2022Review
- Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.Journal of biomedical informatics · 2022Review
- Evaluation of a Digital Health Initiative in Illicit Substance Use: Cross-sectional Survey Study.Journal of medical Internet research · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectiveToolkits are an important knowledge translation strategy for implementing digital health. We studied how toolkits for the implementation and evaluation of digital health were developed, tested, and reported. MATERIALS AND
methodsWe conducted a systematic review of toolkits that had been used, field tested or evaluated in practice, and published in the English language from 2009 to July 2019. We searched several electronic literature sources to identify both peer-reviewed and gray literature, and records were screened as per systematic review conventions.
resultsThirteen toolkits were eventually identified, all of which were developed in North America, Europe, or Australia. All reported their intended purpose, as well as their development process. Eight of the 13 toolkits involved a literature review, 3 did not, and 2 were unclear. Twelve reported an underlying conceptual framework, theory, or model: 3 cited the normalization process theory and 3 others cited the World Health Organization and International Telecommunication Union eHealth Strategy. Seven toolkits were reportedly evaluated, but details were unavailable. Forty-three toolkits were excluded for lack of field-testing. DISCUSSION: Despite a plethora of published toolkits, few were tested, and even fewer were evaluated. Methodological rigor was of concern, as several did not include an underlying conceptual framework, literature review, or evaluation and refinement in real-world settings. Reporting was often inconsistent and unclear, and toolkits rarely reported being evaluated.
conclusionGreater attention needs to be paid to rigor and reporting when developing, evaluating, and reporting toolkits for implementing and evaluating digital health so that they can effectively function as a knowledge translation strategy.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.