ArticleInternational journal of epidemiology2016
Machine learning to assist risk-of-bias assessments in systematic reviews.
Article in International journal of epidemiology, 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 28 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
28 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis.Journal of the American Medical Informatics Association : JAMIA · 2022Pooled it
- Data extraction methods for systematic review (semi)automation: Update of a living systematic review.F1000Research · 2021Pooled it
- Methodological information extraction from randomized controlled trial publications: a pilot study.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2022Trial
- Validation of an AI-Assisted Framework for Systematic Bias Assessment in Observational Studies.Journal of evaluation in clinical practice · 2026Article
- Automated tools for evidence quality assessment: a scoping review.BMC medical research methodology · 2026Article
- RoBuster-Corpus Annotated With Risk of Bias Text Spans in Randomized Controlled Trials in Physiotherapy and Rehabilitation: Corpus Development and Annotation Study.JMIR formative research · 2026Article
- Zero-shot interpretable biomedical literature appraisal with generative large language models.JAMIA open · 2026Article
- Using a large language model (ChatGPT) to assess risk of bias in randomized controlled trials of medical interventions: protocol for a pilot study of interrater agreement with human reviewers.BMC medical research methodology · 2025Article
- Digital Tools to Support the Systematic Review Process: An Introduction.Journal of evaluation in clinical practice · 2025Review
- Reducing risk of bias in interventional studies during their design and conduct: a scoping review.BMC medical research methodology · 2025Article
- SPIRIT-CONSORT-TM: a corpus for assessing transparency of clinical trial protocol and results publications.Scientific data · 2025Article
- SPIRIT-CONSORT-TM: a corpus for assessing transparency of clinical trial protocol and results publications.medRxiv : the preprint server for health sciences · 2025Article
- Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines.Scientific reports · 2024Article
- Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed.Systematic reviews · 2024Article
- CONSORT-TM: Text classification models for assessing the completeness of randomized controlled trial publications.medRxiv : the preprint server for health sciences · 2024Article
- Machine Learning Methods for Systematic Reviews:: A Rapid Scoping Review.Delaware journal of public health · 2023Article
- Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study.Journal of medical Internet research · 2023Article
- Risk of bias assessment in preclinical literature using natural language processing.Research synthesis methods · 2022Article
- Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews.Environmental evidence · 2022Article
- Is Trauma Focused-Cognitive Behavioral Therapy (TF-CBT) Effective in Reducing Trauma Symptoms among Traumatized Refugee Children? A Systematic Review.Journal of child & adolescent trauma · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
Abstract
backgroundRisk-of-bias assessments are now a standard component of systematic reviews. At present, reviewers need to manually identify relevant parts of research articles for a set of methodological elements that affect the risk of bias, in order to make a risk-of-bias judgement for each of these elements. We investigate the use of text mining methods to automate risk-of-bias assessments in systematic reviews. We aim to identify relevant sentences within the text of included articles, to rank articles by risk of bias and to reduce the number of risk-of-bias assessments that the reviewers need to perform by hand.
methodsWe use supervised machine learning to train two types of models, for each of the three risk-of-bias properties of sequence generation, allocation concealment and blinding. The first model predicts whether a sentence in a research article contains relevant information. The second model predicts a risk-of-bias value for each research article. We use logistic regression, where each independent variable is the frequency of a word in a sentence or article, respectively.
resultsWe found that sentences can be successfully ranked by relevance with area under the receiver operating characteristic (ROC) curve (AUC) > 0.98. Articles can be ranked by risk of bias with AUC > 0.72. We estimate that more than 33% of articles can be assessed by just one reviewer, where two reviewers are normally required.
conclusionsWe show that text mining can be used to assist risk-of-bias assessments.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.