ArticleJMIR mHealth and uHealth2021
Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides.
Article in JMIR mHealth and uHealth, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 11 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
11 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Smartphone Apps for Pulmonary Hypertension: Systematic Search and Content Evaluation.JMIR mHealth and uHealth · 2024Pooled it
- A systematic review of features and content quality of Arabic mental mHealth apps.Frontiers in digital health · 2024Pooled it
- Translation and Validation of the User Version of the Mobile Application Rating Scale Into the Polish Language: Cross-Sectional Methodological Study.JMIR formative research · 2026Article
- Methodological Considerations in Evaluating Mental Health Apps: Ensuring Reliability and Patient Safety.JMIR mHealth and uHealth · 2025Article
- Testing and Iterative Improvement of the CEN ISO/TS 82304-2 Health App Quality Assessment: Pilot Interrater Reliability Study.JMIR formative research · 2025Article
- Development, implementation and evaluation of a digital treatment for adolescents with chronic pain: a protocol for a multi-phase study.Frontiers in digital health · 2025Article
- Mobile Apps for Patients with Peritoneal Dialysis: Systematic App Search and Evaluation.Healthcare (Basel, Switzerland) · 2024Article
- Quality assessment of mHealth apps: a scoping review.Frontiers in health services · 2024Article
- Exploring the factors that affect user experience in mobile-health applications: A text-mining and machine-learning approach.Journal of business research · 2023Article
- Mental, Physical and Socio-Economic Status of Adults Living in Spain during the Late Stages of the State of Emergency Caused by COVID-19.International journal of environmental research and public health · 2022Article
- Effects of COVID-19 Social Distancing Measures in Individuals with Chronic Pain Living in Spain in the Late Stages of the Lockdown.International journal of environmental research and public health · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
2 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThere is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed.
objectiveThe main aim of this study was to study the interrater reliability of a new guide - the Mobile App Development and Assessment Guide (MAG) - and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders.
methodsIn order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability.
resultsOnly a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were "Security" (α=0.78) and "Privacy" (α=0.73). In addition, 2 other categories, "Usability" and "Safety," were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29.
conclusionsThis study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.