ArticleBioinformatics (Oxford, England)2023
An extensive benchmark study on biomedical text generation and mining with ChatGPT.
Article in Bioinformatics (Oxford, England), 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 50 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
50 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Microbial named entity recognition and normalisation for AI-assisted literature review and meta-analysis.Bioinformatics (Oxford, England) · 2026Pooled it
- Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.JAMA · 2025Pooled it
- Data extraction methods for systematic review (semi)automation: Update of a living systematic review.F1000Research · 2021Pooled it
- Benchmarking commercial large language models for gene-disease-phenotype extraction from full-text human genetics literature.Quantitative biology (Beijing, China) · 2026Article
- A data-driven method for research trend analysis in a scientific discipline: Application to the journal of biomedical informatics.Journal of biomedical informatics · 2026Article
- A collaborative large language model for drug analysis.Nature biomedical engineering · 2026Article
- Zero-shot interpretable biomedical literature appraisal with generative large language models.JAMIA open · 2026Article
- Review
- GeneReL: a large language model-powered platform for gene regulatory relationship extraction with community curation.Database : the journal of biological databases and curation · 2026Article
- Reduction of supervision for biomedical knowledge discovery.BMC bioinformatics · 2025Article
- Evaluation of SURUS: a named entity recognition NLP system to extract knowledge from interventional study records.BMC medical research methodology · 2025Article
- NMRExtractor: leveraging large language models to construct an experimental NMR database from open-source scientific publications.Chemical science · 2025Article
- The Accuracy of ChatGPT-4o in Interpreting Chest and Abdominal X-Ray Images.Journal of personalized medicine · 2025Article
- Article
- Benchmarking large language models for biomedical natural language processing applications and recommendations.Nature communications · 2025Article
- Unravelling Convergent Signaling Mechanisms Underlying the Aging-Disease Nexus Using Computational Language Analysis.Current issues in molecular biology · 2025Article
- Comparative analysis of generative LLMs for labeling entities in clinical notes.Genomics & informatics · 2025Review
- LitSumm: large language models for literature summarization of noncoding RNAs.Database : the journal of biological databases and curation · 2025Article
- Aligning Large Language Models with Humans: A Comprehensive Survey of ChatGPT's Aptitude in Pharmacology.Drugs · 2025Article
- ArcTEX-a novel clinical data enrichment pipeline to support real-world evidence oncology studies.Frontiers in digital health · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
motivationIn recent years, the development of natural language process (NLP) technologies and deep learning hardware has led to significant improvement in large language models (LLMs). The ChatGPT, the state-of-the-art LLM built on GPT-3.5 and GPT-4, shows excellent capabilities in general language understanding and reasoning. Researchers also tested the GPTs on a variety of NLP-related tasks and benchmarks and got excellent results. With exciting performance on daily chat, researchers began to explore the capacity of ChatGPT on expertise that requires professional education for human and we are interested in the biomedical domain.
resultsTo evaluate the performance of ChatGPT on biomedical-related tasks, this article presents a comprehensive benchmark study on the use of ChatGPT for biomedical corpus, including article abstracts, clinical trials description, biomedical questions, and so on. Typical NLP tasks like named entity recognization, relation extraction, sentence similarity, question and answering, and document classification are included. Overall, ChatGPT got a BLURB score of 58.50 while the state-of-the-art model had a score of 84.30. Through a series of experiments, we demonstrated the effectiveness and versatility of ChatGPT in biomedical text understanding, reasoning and generation, and the limitation of ChatGPT build on GPT-3.5. AVAILABILITY AND IMPLEMENTATION: All the datasets are available from BLURB benchmark https://microsoft.github.io/BLURB/index.html. The prompts are described in the article.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.