Evidence map›Paper›PMID 36220304›Full record

ArticleBMJ health & care informatics2022

User-centred design for machine learning in health care: a case study from care management.

Martin G Seneviratne, Ron C Li, Meredith Schreier, Daniel Lopez-Martinez, Birju S Patel, Alex Yakubovich, Jonas B Kemp, Eric Loreaux, Paul Gamble, Kristel El-Khoury and 9 more

Open access · goldAbstract read
In one paragraph

Article in BMJ health & care informatics, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 11 papers.

0numbers the graph read from it
0cells of the map it votes in
11citing papers in PubMed
1.2field-weighted citation impact, top 19% of its field
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

11 citing papers in PubMed, 21 citations in OpenAlex.

  1. Observational
  2. Article
  3. Article
  4. Review
  5. Article
  6. Article
  7. Review
  8. [Use of AI-based applications by hospital staff: task profiles and qualification requirements].Bundesgesundheitsblatt, Gesundheitsforschung, Gesundheitsschutz · 2024
    Review
  9. Article
  10. Review
  11. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

19 authors at 3 institutions in 1 country.

Martin G Seneviratne *Research, Google Inc, Mountain View, California, USA martsen@google.com.ORCID http://orcid.org/0000-0003-0435-3738
Ron C Li *Department of Medicine, Stanford University, Stanford, California, USA.
Meredith SchreierResearch, Google Inc, Mountain View, California, USA.
Daniel Lopez-MartinezResearch, Google Inc, Mountain View, California, USA.
Birju S PatelResearch, Google Inc, Mountain View, California, USA.
Alex YakubovichResearch, Google Inc, Mountain View, California, USA.
Jonas B KempResearch, Google Inc, Mountain View, California, USA.
Eric LoreauxResearch, Google Inc, Mountain View, California, USA.
Paul GambleResearch, Google Inc, Mountain View, California, USA.
Kristel El-KhouryResearch, Google Inc, Mountain View, California, USA.
Laura VardoulakisResearch, Google Inc, Mountain View, California, USA.
Doris WongResearch, Google Inc, Mountain View, California, USA.
Janjri DesaiDivision of Pharmacy, Stanford Medicine, Stanford, California, USA.
Jonathan H ChenDepartment of Medicine, Stanford University, Stanford, California, USA.
Keith E MorseDepartment of Medicine, Stanford University, Stanford, California, USA.
N Lance DowningDepartment of Medicine, Stanford University, Stanford, California, USA.
Lutz T FingerResearch, Google Inc, Mountain View, California, USA.
Ming-Jun ChenResearch, Google Inc, Mountain View, California, USA.
Nigam ShahDepartment of Medicine, Stanford University, Stanford, California, USA.ORCID http://orcid.org/0000-0001-9385-7158
Google (United States) · USStanford University · USStanford Medicine · US

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

objectivesFew machine learning (ML) models are successfully deployed in clinical practice. One of the common pitfalls across the field is inappropriate problem formulation: designing ML to fit the data rather than to address a real-world clinical pain point.

methodsWe introduce a practical toolkit for user-centred design consisting of four questions covering: (1) solvable pain points, (2) the unique value of ML (eg, automation and augmentation), (3) the actionability pathway and (4) the model's reward function. This toolkit was implemented in a series of six participatory design workshops with care managers in an academic medical centre.

resultsPain points amenable to ML solutions included outpatient risk stratification and risk factor identification. The endpoint definitions, triggering frequency and evaluation metrics of the proposed risk scoring model were directly influenced by care manager workflows and real-world constraints.

conclusionsIntegrating user-centred design early in the ML life cycle is key for configuring models in a clinically actionable way. This toolkit can guide problem selection and influence choices about the technical setup of the ML problem.

Indexed as

Machine LearningUser-Centered DesignDelivery of Health CareHumansPainWorkflowDecision Support Systems, ClinicalMachine LearningSoftware Design

Identifiers

PMID36220304
PMCPMC9557254
OpenAlexW4304757237

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.