Evidence map›Paper›PMID 33683212›Full record

ArticleJournal of medical Internet research2021

Natural Language Processing and Machine Learning for Identifying Incident Stroke From Electronic Health Records: Algorithm Development and Validation.

Yiqing Zhao, Sunyang Fu, Suzette J Bielinski, Paul A Decker, Alanna M Chamberlain, Veronique L Roger, Hongfang Liu, Nicholas B Larson

Abstract read
In one paragraph

Article in Journal of medical Internet research, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers, 3 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
22citing papers in PubMed, 3 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

22 citing papers in PubMed, 3 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Review
  5. Validation of an Algorithm to Identify End-Stage Kidney Disease in Electronic Health Records Data.American journal of kidney diseases : the official journal of the National Kidney Foundation · 2025
    Article
  6. Article
  7. Article
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Review
  14. Article
  15. Positive Predictive Value ofJournal of the American Heart Association · 2023
    Article
  16. Review
  17. Article
  18. Article
  19. Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

8 authors.

Yiqing ZhaoDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0003-2874-8136
Sunyang FuDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0003-1691-5179
Suzette J BielinskiDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0002-2905-5430
Paul A DeckerDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0002-3756-4227
Alanna M ChamberlainDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0003-1888-9584
Veronique L RogerDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0002-9347-7865
Hongfang LiuDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0003-2570-3741
Nicholas B LarsonDepartment of Health Sciences Research, Mayo Clinic, Rochester, MN, United States.ORCID 0000-0002-3468-4215

Funding

Rochester Epidemiology ProjectR01AG034676 · NIA · MAYO CLINIC ROCHESTER · PI ROCCA, WALTER A, ST SAUVER, JENNIFER LYNN · 2010 to 2019
$9.5M
Harnessing the Electronic Health Record to Predict Risk of Cardiovascular DiseaseR01HL136659 · NHLBI · MAYO CLINIC ROCHESTER · PI BIELINSKI, SUZETTE JANINE · 2018 to 2021
$3.2M
Multimorbidity, Rurality, and Outcomes in Atrial Fibrillation in the CommunityR21AG062580 · NIA · MAYO CLINIC ROCHESTER · PI CHAMBERLAIN, ALANNA M · 2019 to 2020
$451k
NHLBI NIH HHS R01 HL136659NIA NIH HHS R01 AG034676NIA NIH HHS R21 AG062580
6 · The paper itself

Abstract

backgroundStroke is an important clinical outcome in cardiovascular research. However, the ascertainment of incident stroke is typically accomplished via time-consuming manual chart abstraction. Current phenotyping efforts using electronic health records for stroke focus on case ascertainment rather than incident disease, which requires knowledge of the temporal sequence of events.

objectiveThe aim of this study was to develop a machine learning-based phenotyping algorithm for incident stroke ascertainment based on diagnosis codes, procedure codes, and clinical concepts extracted from clinical notes using natural language processing.

methodsThe algorithm was trained and validated using an existing epidemiology cohort consisting of 4914 patients with atrial fibrillation (AF) with manually curated incident stroke events. Various combinations of feature sets and machine learning classifiers were compared. Using a heuristic rule based on the composition of concepts and codes, we further detected the stroke subtype (ischemic stroke/transient ischemic attack or hemorrhagic stroke) of each identified stroke. The algorithm was further validated using a cohort (n=150) stratified sampled from a population in Olmsted County, Minnesota (N=74,314).

resultsAmong the 4914 patients with AF, 740 had validated incident stroke events. The best-performing stroke phenotyping algorithm used clinical concepts, diagnosis codes, and procedure codes as features in a random forest classifier. Among patients with stroke codes in the general population sample, the best-performing model achieved a positive predictive value of 86% (43/50; 95% CI 0.74-0.93) and a negative predictive value of 96% (96/100). For subtype identification, we achieved an accuracy of 83% in the AF cohort and 80% in the general population sample.

conclusionsWe developed and validated a machine learning-based algorithm that performed well for identifying incident stroke and for determining type of stroke. The algorithm also performed well on a sample from a general population, further demonstrating its generalizability and potential for adoption by other institutions.

Indexed as

Natural Language ProcessingStrokeAlgorithmsElectronic Health RecordsHumansMachine Learningelectronic health recordsmachine learningnatural language processingstroke

Identifiers

PMID33683212
PMCPMC7985804

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.