ArticleJournal of the American Medical Informatics Association : JAMIA
Towards automatic recognition of scientifically rigorous clinical research evidence.
Article in Journal of the American Medical Informatics Association : JAMIA. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 46 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
46 citing papers in PubMed, 1 synthesis or guideline pooled it.
- A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2010Pooled it
- Article
- Zero-shot interpretable biomedical literature appraisal with generative large language models.JAMIA open · 2026Article
- Boosting efficiency in a clinical literature surveillance system with LightGBM.PLOS digital health · 2024Article
- The McMaster Health Information Research Unit: Over a Quarter-Century of Health Informatics Supporting Evidence-Based Medicine.Journal of medical Internet research · 2024Article
- Automated Category and Trend Analysis of Scientific Articles on Ophthalmology Using Large Language Models: Development and Usability Study.JMIR formative research · 2024Article
- BIR: Biomedical Information Retrieval System for Cancer Treatment in Electronic Health Record Using Transformers.Sensors (Basel, Switzerland) · 2023Article
- Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records.Systematic reviews · 2023Article
- Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study.Journal of medical Internet research · 2023Article
- A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation.JMIR research protocols · 2021Article
- Machine Learning Approaches to Retrieve High-Quality, Clinically Relevant Evidence From the Biomedical Literature: Systematic Review.JMIR medical informatics · 2021Review
- Artificial Intelligence Clinical Evidence Engine for Automatic Identification, Prioritization, and Extraction of Relevant Clinical Oncology Research.JCO clinical cancer informatics · 2021Article
- Refining Boolean queries to identify relevant studies for systematic review updates.Journal of the American Medical Informatics Association : JAMIA · 2020Article
- Clinical Context-Aware Biomedical Text Summarization Using Deep Neural Network: Model Development and Validation.Journal of medical Internet research · 2020Article
- Impact of Automatic Query Generation and Quality Recognition Using Deep Learning to Curate Evidence From Biomedical Literature: Empirical Study.JMIR medical informatics · 2019Article
- A High Recall Classifier for Selecting Articles for MEDLINE Indexing.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2019Article
- Automatic identification of recent high impact clinical articles in PubMed to support clinical decision making using time-agnostic features.Journal of biomedical informatics · 2019Article
- Biomedical text mining for research rigor and integrity: tasks, challenges, directions.Briefings in bioinformatics · 2018Article
- Development and validation of the PEPPER framework (Prenatal Exposure PubMed ParsER) with applications to food additives.Journal of the American Medical Informatics Association : JAMIA · 2018Article
- A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study.Journal of medical Internet research · 2018Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
Abstract
The growing numbers of topically relevant biomedical publications readily available due to advances in document retrieval methods pose a challenge to clinicians practicing evidence-based medicine. It is increasingly time consuming to acquire and critically appraise the available evidence. This problem could be addressed in part if methods were available to automatically recognize rigorous studies immediately applicable in a specific clinical situation. We approach the problem of recognizing studies containing useable clinical advice from retrieved topically relevant articles as a binary classification problem. The gold standard used in the development of PubMed clinical query filters forms the basis of our approach. We identify scientifically rigorous studies using supervised machine learning techniques (Naïve Bayes, support vector machine (SVM), and boosting) trained on high-level semantic features. We combine these methods using an ensemble learning method (stacking). The performance of learning methods is evaluated using precision, recall and F(1) score, in addition to area under the receiver operating characteristic (ROC) curve (AUC). Using a training set of 10,000 manually annotated MEDLINE citations, and a test set of an additional 2,000 citations, we achieve 73.7% precision and 61.5% recall in identifying rigorous, clinically relevant studies, with stacking over five feature-classifier combinations and 82.5% precision and 84.3% recall in recognizing rigorous studies with treatment focus using stacking over word + metadata feature vector. Our results demonstrate that a high quality gold standard and advanced classification methods can help clinicians acquire best evidence from the medical literature.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.