ArticleJournal of the American Medical Informatics Association : JAMIA2022
Combining human and machine intelligence for clinical trial eligibility querying.
Article in Journal of the American Medical Informatics Association : JAMIA, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 19 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
19 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Characterisation of digital therapeutic clinical trials: a systematic review with natural language processing.The Lancet. Digital health · 2024Pooled it
- EC2Seq2Sql: Patient-trial matching with LLM agents.PloS one · 2026Article
- Artificial intelligence in clinical trial participant recruitment and retention: A scoping review and meta-analysis.Journal of clinical and translational science · 2026Review
- Computational Framework for Structuring and Analyzing Clinical Trial Criteria for AI-Guided Fine-grained Matching.Journal of medical systems · 2025Article
- CLEAR: A vision to support clinical evidence lifecycle with continuous learning.Journal of biomedical informatics · 2025Article
- Enhancing Patient-Trial Matching With Large Language Models: A Scoping Review of Emerging Applications and Approaches.JCO clinical cancer informatics · 2025Article
- Illuminating the landscape of high-level clinical trial opportunities in the All of Us Research Program.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- PRISM: Patient Records Interpretation for Semantic clinical trial Matching system using large language models.NPJ digital medicine · 2024Article
- Bridging Data Models in Health Care With a Novel Intermediate Query Format for Feasibility Queries: Mixed Methods Study.JMIR medical informatics · 2024Article
- Distilling large language models for matching patients to clinical trials.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- Criteria2Query 3.0: Leveraging generative large language models for clinical trial eligibility query generation.Journal of biomedical informatics · 2024Article
- Sociotechnical feasibility of natural language processing-driven tools in clinical trial eligibility prescreening for Alzheimer's disease and related dementias.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- Clinical researchers' insights on key data for eligibility screening in clinical studies.Journal of clinical and translational science · 2024Article
- LeafAI: query generator for clinical cohort discovery rivaling a human programmer.Journal of the American Medical Informatics Association : JAMIA · 2023Article
- Piloting an automated clinical trial eligibility surveillance and provider alert system based on artificial intelligence and standard data models.BMC medical research methodology · 2023Observational
- Clinical research staff perceptions on a natural language processing-driven tool for eligibility prescreening: An iterative usability assessment.International journal of medical informatics · 2023Article
- Principal Investigators' Perceptions on Factors Associated with Successful Recruitment in Clinical Trials.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2023Article
- Natural language processing in Alzheimer's disease research: Systematic review of methods, data, and efficacy.Alzheimer's & dementia (Amsterdam, Netherlands)Review
- Development and application of artificial intelligence follow-up system in patients with coronary heart disease.Digital healthArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
Abstract
objectiveTo combine machine efficiency and human intelligence for converting complex clinical trial eligibility criteria text into cohort queries. MATERIALS AND
methodsCriteria2Query (C2Q) 2.0 was developed to enable real-time user intervention for criteria selection and simplification, parsing error correction, and concept mapping. The accuracy, precision, recall, and F1 score of enhanced modules for negation scope detection, temporal and value normalization were evaluated using a previously curated gold standard, the annotated eligibility criteria of 1010 COVID-19 clinical trials. The usability and usefulness were evaluated by 10 research coordinators in a task-oriented usability evaluation using 5 Alzheimer's disease trials. Data were collected by user interaction logging, a demographic questionnaire, the Health Information Technology Usability Evaluation Scale (Health-ITUES), and a feature-specific questionnaire.
resultsThe accuracies of negation scope detection, temporal and value normalization were 0.924, 0.916, and 0.966, respectively. C2Q 2.0 achieved a moderate usability score (3.84 out of 5) and a high learnability score (4.54 out of 5). On average, 9.9 modifications were made for a clinical study. Experienced researchers made more modifications than novice researchers. The most frequent modification was deletion (5.35 per study). Furthermore, the evaluators favored cohort queries resulting from modifications (score 4.1 out of 5) and the user engagement features (score 4.3 out of 5). DISCUSSION AND
conclusionFeatures to engage domain experts and to overcome the limitations in automated machine output are shown to be useful and user-friendly. We concluded that human-computer collaboration is key to improving the adoption and user-friendliness of natural language processing.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.