Evidence map›Paper›PMID 34729675›Full record

ArticleJournal of medical systems2021

Machine Learning for Health: Algorithm Auditing & Quality Control.

Luis Oala, Andrew G Murchison, Pradeep Balachandran, Shruti Choudhary, Jana Fehr, Alixandro Werneck Leite, Peter G Goldschmidt, Christian Johner, Elora D M Schörverth, Rose Nakasi and 29 more

Open access · hybridAbstract readEditorial
In one paragraph

Article in Journal of medical systems, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 1 of them a synthesis that pooled it.

0numbers the graph read from it
0cells of the map it votes in
21citing papers in PubMed, 1 pooled it
2.1field-weighted citation impact, top 11% of its field
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

21 citing papers in PubMed, 1 synthesis or guideline pooled it, 45 citations in OpenAlex.

  1. Pooled it
  2. Explainable AI needs formalization.NPJ artificial intelligence · 2026
    Review
  3. Article
  4. Article
  5. Review
  6. Article
  7. Article
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework.Journal of the American Medical Informatics Association : JAMIA · 2023
    Review
  14. Review
  15. Article
  16. Review
  17. Article
  18. Article
  19. Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

39 authors at 20 institutions in 10 countries.

Luis OalaFraunhofer HHI, Berlin, Germany. luis.oala@hhi.fraunhofer.de.
Andrew G MurchisonOxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom.
Pradeep BalachandranTechnical Consultant (Digital Health), Thiruvananthapuram, India.
Shruti ChoudharyUniversity of Oxford, Oxford, United Kingdom.
Jana FehrHasso-Plattner-Institute of Digital Engineering, Potsdam, Germany.
Alixandro Werneck LeiteMachine Learning Laboratory in Finance and Organizations, Universidade de Brasília, Brasília, Brazil.
Peter G GoldschmidtWorld Development Group Inc, Bethesda, MD, USA.
Christian JohnerJohner Institute, Konstanz, Germany.
Elora D M SchörverthFraunhofer HHI, Berlin, Germany.
Rose NakasiMakerere University, Kampala, Uganda.
Martin MeyerSiemens Healthineers, Erlangen, Germany.
Federico CabitzaUniversity of Milano-Bicocca, Milan, Italy.
Pat BairdPhilips, New Kensington, USA.
Carolin PrabhuOffice of the Auditor General of Norway, Oslo, Norway.
Eva WeickenFraunhofer HHI, Berlin, Germany.
Xiaoxuan LiuUniversity Hospitals Birmingham NHS Foundation Trust & Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom.
Markus WenzelFraunhofer HHI, Berlin, Germany.
Steffen VoglerBayer AG, Berlin, Germany.
Darlington AkogominoHealth AI Labs, Accra, Ghana.
Shada AlsalamahInformation Systems Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia.
Emre KazimUniversity College London, London, United Kingdom.
Adriano KoshiyamaUniversity College London, London, United Kingdom.
Sven PiechottkaOpen Regulatory, Bonn, Germany.
Sheena MacphersonMIOTIFY LTD, London, United Kingdom.
Ian ShadforthMIOTIFY LTD, London, United Kingdom.
Regina GeierhoferIEC TC62 and Siemens Healthineers, Erlangen, Germany.
Christian MatekHelmholtz Zentrum München, Neuherberg, Germany.
Joachim KroisOral Diagnostics Digital Health Health Services Research, Charité-Universitätsmedizin, Berlin, Germany.
Bruno SanguinettiDotphoton AG, Zug, Switzerland.
Matthew ArentzDepartment of Global Health, University of Washington, Washington, USA.
Pavol BielikLatticeFlow & ETH Zurich, Zürich, Switzerland.
Saul Calderon-RamirezDe Montfort University & Instituto Tecnologico de Costa Rica, Cartago, Costa Rica.
Auss AbboodRobert Koch Institut, Berlin, Germany.
Nicolas LangerDepartment of Psychology, University of Zurich, Zürich, Switzerland.
Stefan HaufeTechnische Universität Berlin, Berlin, Germany.
Ferath KherifLaboratory for Research in Neuroimaging, Department of Clinical Neuroscience, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland.
Sameer PujariDigital Health and Innovation Department, Science Division, World Health Organization, Winterthur, Switzerland.
Wojciech SamekFraunhofer HHI, Berlin, Germany.
Thomas WiegandFraunhofer HHI, Berlin, Germany.
Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute · DESiemens Healthineers (Germany) · DEUniversity College London · GBBayer (Germany) · DECharité - Universitätsmedizin Berlin · DEETH Zurich · CHHasso Plattner Institute · DEHelmholtz Zentrum München · DEInstituto Tecnológico de Costa Rica · CRJohner Institut (Germany) · DEKing Saud University · SAMakerere University · UGOffice of the Auditor General of Norway · NOOxford University Hospitals NHS Trust · GBPhilips (United States) · USRobert Koch Institute · DETechnische Universität Berlin · DEUniversidade de Brasília · BRUniversity Hospitals Birmingham NHS Foundation Trust · GBUniversity of Lausanne · CH

Funding

World Health Organization 001
6 · The paper itself

Abstract

Developers proposing new machine learning for health (ML4H) tools often pledge to match or even surpass the performance of existing tools, yet the reality is usually more complicated. Reliable deployment of ML4H to the real world is challenging as examples from diabetic retinopathy or Covid-19 screening show. We envision an integrated framework of algorithm auditing and quality control that provides a path towards the effective and reliable application of ML systems in healthcare. In this editorial, we give a summary of ongoing work towards that vision and announce a call for participation to the special issue  Machine Learning for Health: Algorithm Auditing & Quality Control in this journal to advance the practice of ML4H auditing.

Indexed as

AlgorithmsMachine LearningQuality ControlHumansAlgorithmArtificial intelligenceAuditingHealthMachine learningQuality control

Identifiers

PMID34729675
PMCPMC8562935
OpenAlexW3208920975

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.