ArticleJMIR mHealth and uHealth2016
Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps.
Article in JMIR mHealth and uHealth, 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 56 papers, 5 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
56 citing papers in PubMed, 5 syntheses or guidelines pooled it.
- Using Health and Well-Being Apps for Behavior Change: A Systematic Search and Rating of Apps.JMIR mHealth and uHealth · 2019Pooled it
- Evaluation of Mobile Apps Targeted to Parents of Infants in the Neonatal Intensive Care Unit: Systematic App Review.JMIR mHealth and uHealth · 2019Pooled it
- The App Behavior Change Scale: Creation of a Scale to Assess the Potential of Apps to Promote Behavior Change.JMIR mHealth and uHealth · 2019Pooled it
- Criteria for assessing the quality of mHealth apps: a systematic review.Journal of the American Medical Informatics Association : JAMIA · 2018Pooled it
- Social anxiety apps: a systematic review and assessment of app descriptors across mobile store platforms.Evidence-based mental health · 2017Pooled it
- Efficacy of a Web-Based Guided Recommendation Service for a Curated List of Readily Available Mental Health and Well-Being Mobile Apps for Young People: Randomized Controlled Trial.Journal of medical Internet research · 2017Trial
- Functionality and quality assessment of Iranian nutrition-related apps using the mobile app rating scale.Journal of education and health promotion · 2026Article
- Evaluation of medical education mobile apps with MARS and MARuL in a TPACK informed model.Scientific reports · 2025Article
- Automated framework for comprehensive usability analysis of healthcare websites using web parsing.Scientific reports · 2025Article
- Testing and Iterative Improvement of the CEN ISO/TS 82304-2 Health App Quality Assessment: Pilot Interrater Reliability Study.JMIR formative research · 2025Article
- A Structured Review of Commercially Available Cardiac Rehabilitation mHealth Applications Using the Mobile Application Rating Scale.Journal of cardiopulmonary rehabilitation and prevention · 2022Article
- A Novel Method for Evaluating Mobile Apps (App Rating Inventory): Development Study.JMIR mHealth and uHealth · 2022Review
- Review of Mobile Apps for Women With Anxiety in Pregnancy: Maternity Care Professionals' Guide to Locating and Assessing Anxiety Apps.Journal of medical Internet research · 2022Review
- The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective.Healthcare (Basel, Switzerland) · 2022Article
- mHealth Solutions for Perinatal Mental Health: Scoping Review and Appraisal Following the mHealth Index and Navigation Database Framework.JMIR mHealth and uHealth · 2022Article
- Do pain management apps use evidence-based psychological components? A systematic review of app content and quality.Canadian journal of pain = Revue canadienne de la douleur · 2022Review
- Quality evaluation of stress, anxiety and depression apps for COVID-19.Journal of affective disorders reports · 2021Article
- Assessing Apps for Health Care Workers Using the ISYScore-Pro Scale: Development and Validation Study.JMIR mHealth and uHealth · 2021Article
- User Engagement and Usability of Suicide Prevention Apps: Systematic Search in App Stores and Content Analysis.JMIR formative research · 2021Article
- Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers.Journal of the American Medical Informatics Association : JAMIA · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThere are over 165,000 mHealth apps currently available to patients, but few have undergone an external quality review. Furthermore, no standardized review method exists, and little has been done to examine the consistency of the evaluation systems themselves.
objectiveWe sought to determine which measures for evaluating the quality of mHealth apps have the greatest interrater reliability.
methodsWe identified 22 measures for evaluating the quality of apps from the literature. A panel of 6 reviewers reviewed the top 10 depression apps and 10 smoking cessation apps from the Apple iTunes App Store on these measures. Krippendorff's alpha was calculated for each of the measures and reported by app category and in aggregate.
resultsThe measure for interactiveness and feedback was found to have the greatest overall interrater reliability (alpha=.69). Presence of password protection (alpha=.65), whether the app was uploaded by a health care agency (alpha=.63), the number of consumer ratings (alpha=.59), and several other measures had moderate interrater reliability (alphas>.5). There was the least agreement over whether apps had errors or performance issues (alpha=.15), stated advertising policies (alpha=.16), and were easy to use (alpha=.18). There were substantial differences in the interrater reliabilities of a number of measures when they were applied to depression versus smoking apps.
conclusionsWe found wide variation in the interrater reliability of measures used to evaluate apps, and some measures are more robust across categories of apps than others. The measures with the highest degree of interrater reliability tended to be those that involved the least rater discretion. Clinical quality measures such as effectiveness, ease of use, and performance had relatively poor interrater reliability. Subsequent research is needed to determine consistent means for evaluating the performance of apps. Patients and clinicians should consider conducting their own assessments of apps, in conjunction with evaluating information from reviews.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.