Evidence map›Paper›PMID 33059590›Full record

ArticleBMC medical research methodology2020

An evaluation of DistillerSR's machine learning-based prioritization tool for title/abstract screening - impact on reviewer-relevant outcomes.

C Hamel, S E Kelly, K Thavorn, D B Rice, G A Wells, B Hutton

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in BMC medical research methodology, 2020. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 65 papers, 5 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
65citing papers in PubMed, 5 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

65 citing papers in PubMed, 5 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Article
  7. Review
  8. Review
  9. Article
  10. Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
  16. Article
  17. Article
  18. Review
  19. Article
  20. Article

5 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

6 authors.

C HamelClinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Box 201b, Ottawa, Ontario, K1H 8L6, Canada. cahamel@ohri.ca.ORCID 0000-0002-5871-2137
S E KellyCardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Ontario, Canada.
K ThavornClinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Box 201b, Ottawa, Ontario, K1H 8L6, Canada.
D B RiceClinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Box 201b, Ottawa, Ontario, K1H 8L6, Canada.
G A WellsClinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Box 201b, Ottawa, Ontario, K1H 8L6, Canada.
B HuttonClinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Box 201b, Ottawa, Ontario, K1H 8L6, Canada.

Funding

CIHR
6 · The paper itself

Abstract

backgroundSystematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings.

methodsUsing a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented.

resultsAmong the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%).

conclusionsThe prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.

Indexed as

Artificial IntelligenceMachine LearningAlgorithmsHumansMass ScreeningProspective StudiesArtificial intelligenceAutomationEfficiencyMachine learningNatural language processingPrioritizationRapid reviewsSystematic reviewsTime savingsTrue recall

Identifiers

PMID33059590
PMCPMC7559198

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.