Evidence map›Paper›PMID 33982356›Full record

SynthesisJournal of evaluation in clinical practice2021

Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems.

Marta Weronika Wronikowska, James Malycha, Lauren J Morgan, Verity Westgate, Tatjana Petrinic, J Duncan Young, Peter J Watkinson

Abstract readSystematic Review
PubMed Full text at PMC Publisher
In one paragraph

Synthesis in Journal of evaluation in clinical practice, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 26 papers, 5 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
26citing papers in PubMed, 5 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

26 citing papers in PubMed, 5 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Guideline
  4. Guideline
  5. Pooled it
  6. Article
  7. Article
  8. Article
  9. Article
  10. Review
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
  16. Article
  17. Review
  18. Article
  19. Review
  20. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

7 authors.

Marta Weronika WronikowskaCritical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.ORCID 0000-0002-5416-7872
James MalychaCritical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
Lauren J MorganNuffield Department of Surgical Sciences, University of Oxford, John Radcliffe Hospital, Oxford, UK.
Verity WestgateCritical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
Tatjana PetrinicBodleian Health Care Libraries, John Radcliffe Hospital, University of Oxford, Oxford, UK.
J Duncan YoungCritical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
Peter J WatkinsonCritical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.

Funding

Department of HealthWellcome TrustWellcome Trust WT-103703/Z/14/Z
6 · The paper itself

Abstract

BACKGROUND AND

objectivesElectronic healthcare records have become central to patient care. Evaluation of new systems include a variety of usability evaluation methods or usability metrics (often referred to interchangeably as usability components or usability attributes). This study reviews the breadth of usability evaluation methods, metrics, and associated measurement techniques that have been reported to assess systems designed for hospital staff to assess inpatient clinical condition.

methodsFollowing Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology, we searched Medline, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, and Open Grey from 1986 to 2019. For included studies, we recorded usability evaluation methods or usability metrics as appropriate, and any measurement techniques applied to illustrate these. We classified and described all usability evaluation methods, usability metrics, and measurement techniques. Study quality was evaluated using a modified Downs and Black checklist.

resultsThe search identified 1336 studies. After abstract screening, 130 full texts were reviewed. In the 51 included studies 11 distinct usability evaluation methods were identified. Within these usability evaluation methods, seven usability metrics were reported. The most common metrics were ISO9241-11 and Nielsen's components. An additional "usefulness" metric was reported in almost 40% of included studies. We identified 70 measurement techniques used to evaluate systems. Overall study quality was reflected in a mean modified Downs and Black checklist score of 6.8/10 (range 1-9) 33% studies classified as "high-quality" (scoring eight or higher), 51% studies "moderate-quality" (scoring 6-7), and the remaining 16% (scoring below five) were "low-quality."

conclusionThere is little consistency within the field of electronic health record systems evaluation. This review highlights the variability within usability methods, metrics, and reporting. Standardized processes may improve evaluation and comparison electronic health record systems and improve their development and implementation.

Indexed as

BenchmarkingTelemedicineElectronicsHospitalsHumansSoftwareelectronic health recordselectronic patients record (EPR)systematic reviewusability methodsusability metrics

Identifiers

PMID33982356
PMCPMC9438452

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.