Evidence map›Paper›PMID 37061711›Full record

ArticleSystematic reviews2023

Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study.

Ana Helena Salles Dos Reis, Ana Luiza Miranda de Oliveira, Carolina Fritsch, James Zouch, Paulo Ferreira, Janaine Cunha Polese

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Systematic reviews, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 4 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
21citing papers in PubMed, 4 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

21 citing papers in PubMed, 4 syntheses or guidelines pooled it.

  1. Prevalence of Stroke-Related Sarcopenia: A Systematic Review and Meta-Analysis.Physiotherapy research international : the journal for researchers and clinicians in physical therapy · 2026
    Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Review
  6. Article
  7. Article
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
  16. Article
  17. Article
  18. Article
  19. Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

6 authors.

Ana Helena Salles Dos ReisPost-Graduate Program of Health Sciences, Faculdade Ciências Médicas de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
Ana Luiza Miranda de OliveiraPost-Graduate Program of Health Sciences, Faculdade Ciências Médicas de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
Carolina FritschFaculty of Medicine and Health, School of Health Sciences, Sydney Musculoskeletal Health, The Kolling Institute, The University of Sydney, Sydney, NSW, Australia.
James ZouchFaculty of Health Sciences, The University of Sydney, Sydney, NSW, Australia.
Paulo Ferreira *Faculty of Health Sciences, The University of Sydney, Sydney, NSW, Australia.
Janaine Cunha Polese *Post-Graduate Program of Health Sciences, Faculdade Ciências Médicas de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil. janaine.polese@cienciasmedicasmg.edu.br.ORCID 0000-0003-3366-1545

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

objectiveTo investigate the usefulness and performance metrics of three freely-available softwares (Rayyan®, Abstrackr® and Colandr®) for title screening in systematic reviews. STUDY DESIGN AND

settingIn this methodological study, the usefulness of softwares to screen titles in systematic reviews was investigated by the comparison between the number of titles identified by software-assisted screening and those by manual screening using a previously published systematic review. To test the performance metrics, sensitivity, specificity, false negative rate, proportion missed, workload and timing savings were calculated. A purposely built survey was used to evaluate the rater's experiences regarding the softwares' performances.

resultsRayyan® was the most sensitive software and raters correctly identified 78% of the true positives. All three softwares were specific and raters correctly identified 99% of the true negatives. They also had similar values for precision, proportion missed, workload and timing savings. Rayyan®, Abstrackr® and Colandr® had 21%, 39% and 34% of false negatives rates, respectively. Rayyan presented the best performance (35/40) according to the raters.

conclusionRayyan®, Abstrackr® and Colandr® are useful tools and provided good metric performance results for systematic title screening. Rayyan® appears to be the best ranked on the quantitative and on the raters' perspective evaluation. The most important finding of this study is that the use of software to screen titles does not remove any title that would meet the inclusion criteria for the final review, being valuable resources to facilitate the screening process.

Indexed as

Machine LearningSoftwareHumansSystematic Reviews as TopicWorkloadCitation screeningMachine learningSoftware toolsText miningUser Experience

Identifiers

PMID37061711
PMCPMC10105467

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.