ArticleJMIR medical informatics2015
Benchmarking clinical speech recognition and information extraction: new data, methods, and evaluations.
Article in JMIR medical informatics, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 19 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
19 citing papers in PubMed, 2 syntheses or guidelines pooled it, 56 citations in OpenAlex.
- Natural Language Processing Applications in the Clinical Neurosciences: A Machine Learning Augmented Systematic Review.Acta neurochirurgica. Supplement · 2022Pooled it
- Speech recognition for clinical documentation from 1990 to 2018: a systematic review.Journal of the American Medical Informatics Association : JAMIA · 2019Pooled it
- AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models.JMIR nursing · 2026Article
- Benefits, barriers, and facilitators of using speech recognition technology in nursing documentation and reporting: A cross-sectional study.Health science reports · 2023Article
- Exploring nurses' online perspectives and social networks during a global pandemic COVID-19.Public health nursing (Boston, Mass.) · 2022Article
- Cognitive Computing-Based CDSS in Medical Practice.Health data science · 2021Review
- Reliability of Supervised Machine Learning Using Synthetic Data in Health Care: Model to Preserve Privacy for Data Sharing.JMIR medical informatics · 2020Article
- Automatic Labeled Dialogue Generation for Nursing Record Systems.Journal of personalized medicine · 2020Article
- BioConceptVec: Creating and evaluating literature-based biomedical concept embeddings on a large scale.PLoS computational biology · 2020Article
- Evaluating a Spoken Dialogue System for Recording Systems of Nursing Care.Sensors (Basel, Switzerland) · 2019Article
- Adapting State-of-the-Art Deep Language Models to Clinical Information Extraction Systems: Potentials, Challenges, and Solutions.JMIR medical informatics · 2019Article
- Article
- A multitask bi-directional RNN model for named entity recognition on Chinese electronic medical records.BMC bioinformatics · 2018Article
- Using clinical Natural Language Processing for health outcomes research: Overview and actionable suggestions for future advances.Journal of biomedical informatics · 2018Article
- Advancing the State of the Art in Clinical Natural Language Processing through Shared Tasks.Yearbook of medical informatics · 2018Review
- Scholarly Influence of the Conference and Labs of the Evaluation Forum eHealth Initiative: Review and Bibliometric Study of the 2012 to 2017 Outcomes.JMIR research protocols · 2018Article
- A systematic comparison of contemporary automatic speech recognition engines for conversational clinical speech.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2018Article
- Clinical Natural Language Processing in 2015: Leveraging the Variety of Texts of Clinical Interest.Yearbook of medical informatics · 2016Article
- Natural Language Processing-Enabled and Conventional Data Capture Methods for Input to Electronic Health Records: A Comparative Usability Study.JMIR medical informatics · 2016Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors at 2 institutions in 2 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundOver a tenth of preventable adverse events in health care are caused by failures in information flow. These failures are tangible in clinical handover; regardless of good verbal handover, from two-thirds to all of this information is lost after 3-5 shifts if notes are taken by hand, or not at all. Speech recognition and information extraction provide a way to fill out a handover form for clinical proofing and sign-off.
objectiveThe objective of the study was to provide a recorded spoken handover, annotated verbatim transcriptions, and evaluations to support research in spoken and written natural language processing for filling out a clinical handover form. This dataset is based on synthetic patient profiles, thereby avoiding ethical and legal restrictions, while maintaining efficacy for research in speech-to-text conversion and information extraction, based on realistic clinical scenarios. We also introduce a Web app to demonstrate the system design and workflow.
methodsWe experiment with Dragon Medical 11.0 for speech recognition and CRF++ for information extraction. To compute features for information extraction, we also apply CoreNLP, MetaMap, and Ontoserver. Our evaluation uses cross-validation techniques to measure processing correctness.
resultsThe data provided were a simulation of nursing handover, as recorded using a mobile device, built from simulated patient records and handover scripts, spoken by an Australian registered nurse. Speech recognition recognized 5276 of 7277 words in our 100 test documents correctly. We considered 50 mutually exclusive categories in information extraction and achieved the F1 (ie, the harmonic mean of Precision and Recall) of 0.86 in the category for irrelevant text and the macro-averaged F1 of 0.70 over the remaining 35 nonempty categories of the form in our 101 test documents.
conclusionsThe significance of this study hinges on opening our data, together with the related performance benchmarks and some processing software, to the research and development community for studying clinical documentation and language-processing. The data are used in the CLEFeHealth 2015 evaluation laboratory for a shared task on speech recognition.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.