ArticleApplied clinical informatics2019
Interactive NLP in Clinical Care: Identifying Incidental Findings in Radiology Reports.
Article in Applied clinical informatics, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 14 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
14 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review.Frontiers in psychology · 2022Pooled it
- Automated identification of incidentalomas requiring follow-up: A multi-anatomy evaluation of LLM-based and supervised approaches.Journal of biomedical informatics · 2026Article
- A comprehensive bedside chest radiography dataset with structured, itemized and graded radiologic reports.Scientific data · 2026Article
- Structured Transformation of Unstructured Prostate MRI Reports Using Large Language Models.Tomography (Ann Arbor, Mich.) · 2025Article
- System-driven longitudinal follow-up of incidental imaging findings.The British journal of radiology · 2023Article
- Natural Language Processing Model for Identifying Critical Findings-A Multi-Institutional Study.Journal of digital imaging · 2023Article
- Natural Language Processing and Graph Theory: Making Sense of Imaging Records in a Novel Representation Frame.JMIR medical informatics · 2022Article
- Overview of Noninterpretive Artificial Intelligence Models for Safety, Quality, Workflow, and Education Applications in Radiology Practice.Radiology. Artificial intelligence · 2022Review
- Machine Learning for Detection of Correct Peripherally Inserted Central Catheter Tip Position from Radiology Reports in Infants.Applied clinical informatics · 2021Article
- Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology.JMIR medical informatics · 2021Article
- Translational NLP: A New Paradigm and General Principles for Natural Language Processing Research.Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting · 2021Article
- Different Data Mining Approaches Based Medical Text Data.Journal of healthcare engineering · 2021Review
- A Web Application for Adrenal Incidentaloma Identification, Tracking, and Management Using Machine Learning.Applied clinical informatics · 2020Article
- Interactive Exploration of Longitudinal Cancer Patient Histories Extracted From Clinical Text.JCO clinical cancer informatics · 2020Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
Abstract
backgroundDespite advances in natural language processing (NLP), extracting information from clinical text is expensive. Interactive tools that are capable of easing the construction, review, and revision of NLP models can reduce this cost and improve the utility of clinical reports for clinical and secondary use.
objectivesWe present the design and implementation of an interactive NLP tool for identifying incidental findings in radiology reports, along with a user study evaluating the performance and usability of the tool.
methodsExpert reviewers provided gold standard annotations for 130 patient encounters (694 reports) at sentence, section, and report levels. We performed a user study with 15 physicians to evaluate the accuracy and usability of our tool. Participants reviewed encounters split into intervention (with predictions) and control conditions (no predictions). We measured changes in model performance, the time spent, and the number of user actions needed. The System Usability Scale (SUS) and an open-ended questionnaire were used to assess usability.
resultsStarting from bootstrapped models trained on 6 patient encounters, we observed an average increase in F1 score from 0.31 to 0.75 for reports, from 0.32 to 0.68 for sections, and from 0.22 to 0.60 for sentences on a held-out test data set, over an hour-long study session. We found that tool helped significantly reduce the time spent in reviewing encounters (134.30 vs. 148.44 seconds in intervention and control, respectively), while maintaining overall quality of labels as measured against the gold standard. The tool was well received by the study participants with a very good overall SUS score of 78.67.
conclusionThe user study demonstrated successful use of the tool by physicians for identifying incidental findings. These results support the viability of adopting interactive NLP tools in clinical care settings for a wider range of clinical applications.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.