Evidence map›Paper›PMID 36414832›Full record

ArticleJournal of digital imaging2023

Utilizing a Digital Swarm Intelligence Platform to Improve Consensus Among Radiologists and Exploring Its Applications.

Rutwik Shah, Bruno Astuto Arouche Nunes, Tyler Gleason, Will Fletcher, Justin Banaga, Kevin Sweetwood, Allen Ye, Rina Patel, Kevin McGill, Thomas Link and 3 more

Open access · hybridFull text read
PubMed Full text at PMC Publisher
In one paragraph

Article in Journal of digital imaging, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 2 papers.

0numbers the graph read from it
0cells of the map it votes in
2citing papers in PubMed
0.3field-weighted citation impact, top 48% of its field
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

2 citing papers in PubMed, 2 citations in OpenAlex.

  1. Article
  2. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

13 authors at 2 institutions in 1 country.

Rutwik ShahDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA. rutwik.shah@ucsf.edu.
Bruno Astuto Arouche NunesDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Tyler GleasonDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Will FletcherDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Justin BanagaDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Kevin SweetwoodDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Allen YeDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Rina PatelDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Kevin McGillDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Thomas LinkDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Jason CraneDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Valentina PedoiaDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
Sharmila MajumdarDepartment of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA.
University of California, San Francisco · USUniversity of California System · US

Funding

Biomedical Imaging for Clinician ScientistsT32EB001631 · NIBIB · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI THOMAS M LINK · 2005 to 2026
$5.6M
Understanding the mechanisms by which weight change affects progression of knee osteoarthritis in obese and overweight individuals: An analysis of the Osteoarthritis Initiative DatasetR01AR078917 · NIAMS · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI GABRIELLE JOSEPH, THOMAS M LINK · 2021 to 2026
$3.4M
Deep Learning for Characterizing Knee Joint Degeneration Predicting Progression of Osteoarthritis and Total Knee ReplacementR33AR073552 · NIAMS · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI MAJUMDAR, SHARMILA, PEDOIA, VALENTINA · 2020 to 2020
$404k
NIAMS NIH HHS R01 AR078917NIAMS NIH HHS R33 AR073552NIBIB NIH HHS T32 EB001631
6 · The paper itself

Abstract

Radiologists today play a central role in making diagnostic decisions and labeling images for training and benchmarking artificial intelligence (AI) algorithms. A key concern is low inter-reader reliability (IRR) seen between experts when interpreting challenging cases. While team-based decisions are known to outperform individual decisions, inter-personal biases often creep up in group interactions which limit nondominant participants from expressing true opinions. To overcome the dual problems of low consensus and interpersonal bias, we explored a solution modeled on bee swarms. Two separate cohorts, three board-certified radiologists, (cohort 1), and five radiology residents (cohort 2) collaborated on a digital swarm platform in real time and in a blinded fashion, grading meniscal lesions on knee MR exams. These consensus votes were benchmarked against clinical (arthroscopy) and radiological (senior-most radiologist) standards of reference using Cohen's kappa. The IRR of the consensus votes was then compared to the IRR of the majority and most confident votes of the two cohorts. IRR was also calculated for predictions from a meniscal lesion detecting AI algorithm. The attending cohort saw an improvement of 23% in IRR of swarm votes (k = 0.34) over majority vote (k = 0.11). Similar improvement of 23% in IRR (k = 0.25) in 3-resident swarm votes over majority vote (k = 0.02) was observed. The 5-resident swarm had an even higher improvement of 30% in IRR (k = 0.37) over majority vote (k = 0.07). The swarm consensus votes outperformed individual and majority vote decision in both the radiologists and resident cohorts. The attending and resident swarms also outperformed predictions from a state-of-the-art AI algorithm.

Indexed as

Artificial IntelligenceRadiologistsAnimalsConsensus Statements as TopicHumansIntelligenceReproducibility of ResultsArtificial intelligenceConsensus decisionsInter-rater reliabilitySwarm intelligenceWorkflow tools

Identifiers

PMID36414832
PMCPMC10039189
OpenAlexW3197084127

What OpenQuestion holds

Textfull text, public
LicenceCC BY
measurements read50
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.