ArticleJournal of biomedical informatics2024
Criteria2Query 3.0: Leveraging generative large language models for clinical trial eligibility query generation.
Article in Journal of biomedical informatics, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 23 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
23 citing papers in PubMed.
- From study design to executable code: automating target trial emulation with large language models.JAMIA open · 2026Article
- Empowering clinical trial design with agentic intelligence and real-world data.Nature communications · 2026Article
- A data-driven method for research trend analysis in a scientific discipline: Application to the journal of biomedical informatics.Journal of biomedical informatics · 2026Article
- Article
- The role of AI in oncology: present applications and future horizons.NPJ precision oncology · 2026Review
- A neural-symbolic AI agent system for biomedical concept mapping.NPJ digital medicine · 2026Article
- Artificial intelligence for clinical trial design, conduct, and analysis: a narrative review.ESMO real world data and digital oncology · 2026Review
- Mapping eligibility criteria in oncology target trial emulations using real-world data: a scoping review.BMC medical research methodology · 2026Article
- Artificial intelligence in clinical trial participant recruitment and retention: A scoping review and meta-analysis.Journal of clinical and translational science · 2026Review
- EC2Seq2Sql: Patient-trial matching with LLM agents.PloS one · 2026Article
- Scalable scientific interest profiling using large language models.Journal of biomedical informatics · 2025Article
- Human evaluation of large language models in healthcare: gaps, challenges, and the need for standardization.npj health systems · 2025Article
- Large Language Models for Automating Clinical Trial Criteria Conversion to Observational Medical Outcomes Partnership Common Data Model Queries: Validation and Evaluation Study.JMIR medical informatics · 2025Article
- CLEAR: A vision to support clinical evidence lifecycle with continuous learning.Journal of biomedical informatics · 2025Article
- Enhanced pre-recruitment framework for clinical trial questionnaires through the integration of large language models and knowledge graphs.Scientific reports · 2025Article
- A perspective for adapting generalist AI to specialized medical AI applications and their challenges.NPJ digital medicine · 2025Review
- Enhancing Patient-Trial Matching With Large Language Models: A Scoping Review of Emerging Applications and Approaches.JCO clinical cancer informatics · 2025Article
- Leveraging long context in retrieval augmented language models for medical question answering.NPJ digital medicine · 2025Article
- Hybrid natural language processing tool for semantic annotation of medical texts in Spanish.BMC bioinformatics · 2025Article
- Harnessing the AI/ML in Drug and Biological Products Discovery and Development: The Regulatory Perspective.Pharmaceuticals (Basel, Switzerland) · 2025Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
Abstract
objectiveAutomated identification of eligible patients is a bottleneck of clinical research. We propose Criteria2Query (C2Q) 3.0, a system that leverages GPT-4 for the semi-automatic transformation of clinical trial eligibility criteria text into executable clinical database queries. MATERIALS AND
methodsC2Q 3.0 integrated three GPT-4 prompts for concept extraction, SQL query generation, and reasoning. Each prompt was designed and evaluated separately. The concept extraction prompt was benchmarked against manual annotations from 20 clinical trials by two evaluators, who later also measured SQL generation accuracy and identified errors in GPT-generated SQL queries from 5 clinical trials. The reasoning prompt was assessed by three evaluators on four metrics: readability, correctness, coherence, and usefulness, using corrected SQL queries and an open-ended feedback questionnaire.
resultsOut of 518 concepts from 20 clinical trials, GPT-4 achieved an F1-score of 0.891 in concept extraction. For SQL generation, 29 errors spanning seven categories were detected, with logic errors being the most common (n = 10; 34.48 %). Reasoning evaluations yielded a high coherence rating, with the mean score being 4.70 but relatively lower readability, with a mean of 3.95. Mean scores of correctness and usefulness were identified as 3.97 and 4.37, respectively.
conclusionGPT-4 significantly improves the accuracy of extracting clinical trial eligibility criteria concepts in C2Q 3.0. Continued research is warranted to ensure the reliability of large language models.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.