ArticleJournal of medical Internet research2021
What Every Reader Should Know About Studies Using Electronic Health Record Data but May Be Afraid to Ask.
Abstract read
In one paragraph
Article in Journal of medical Internet research, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 60 papers, 2 of them syntheses that pooled it.
0numbers the graph read from it
0cells of the map it votes in
60citing papers in PubMed, 2 pooled it
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
60 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Electronic health record data quality assessment and tools: a systematic review.Journal of the American Medical Informatics Association : JAMIA · 2023Pooled it
- Digital Health Data Quality Issues: Systematic Review.Journal of medical Internet research · 2023Pooled it
- Classifying Sickle Cell Disease Subtypes from Clinical Reports: Algorithm Validation and ICD-10 Accuracy Assessment in Five French Hospitals.Journal of medical systems · 2026Article
- HealthFlow: automating electronic health record analysis via a strategically self-evolving multi-agent framework.NPJ digital medicine · 2026Article
- A distributed analysis approach for pharmacovigilance data from electronic medical records in German university hospitals: the POLAR_MI ETL Pipeline.BMC medical informatics and decision making · 2026Article
- The Consistency of Hypercapnic Respiratory Failure Case Definitions in Electronic Health Record Data.Chest · 2026Article
- When algorithms infer gender: revisiting computational phenotyping with electronic health records data.Biology of sex differences · 2025Review
- Multimorbidity measurement strategies for predicting hospital visits.BMC health services research · 2025Article
- Data science for pediatric infectious disease: utilizing COVID-19 as a model.Current opinion in infectious diseases · 2025Review
- TriNetX Dataworks-USA: Overview of a Multi-Purpose, De-Identified, Federated Electronic Health Record Real-World Data and Analytics Network and Comparison to the US Census.Pharmacoepidemiology and drug safety · 2025Article
- Predicting Early-Onset Colorectal Cancer in Individuals Below Screening Age Using Machine Learning and Real-World Data: Case Control Study.JMIR cancer · 2025Article
- Utilization of Computable Phenotypes in Electronic Health Record Research: A Review and Case Study in Atopic Dermatitis.The Journal of investigative dermatology · 2025Review
- Transparency in the secondary use of health data: assessing the status quo of guidance and best practices.Royal Society open science · 2025Article
- Breaking ICD Codes: Identifying Ambiguous Respiratory Infection Codes via Regional Diagnosis Heterogeneity.Annals of family medicine · 2025Article
- With big data comes big responsibility: Strategies for utilizing aggregated, standardized, de-identified electronic health record data for research.Clinical and translational science · 2025Article
- Clinical Phenotypes May be Able to Identify Populations With Nonalcoholic Fatty Liver-Spectrum Disease.Gastro hep advances · 2025Article
- "Goldmine" or "big mess"? An interview study on the challenges of designing, operating, and ensuring the durability of Clinical Data Warehouses in France and Belgium.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- The role and limitations of electronic medical records versus patient interviews for determining symptoms of, underlying comorbidities of, and medication use by patients with COVID-19.American journal of epidemiology · 2024Article
- Implementation of the Observational Medical Outcomes Partnership Model in Electronic Medical Record Systems: Evaluation Study Using Factor Analysis and Decision-Making Trial and Evaluation Laboratory-Best-Worst Methods.JMIR medical informatics · 2024Article
- Comorbidity Profiles of Posttraumatic Stress Disorder Across the Medical Phenome.Biological psychiatry global open science · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
40 authors.
Isaac S KohaneDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0003-2192-5160
Bruce J AronowBiomedical Informatics, Cincinnati Children's Hospital Medical Center, University of Cincinnati, Cincinnati, OH, United States.ORCID 0000-0001-5109-6514
Paul AvillachDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-0235-7543
Brett K Beaulieu-JonesDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-6700-1468
Riccardo BellazziDepartment of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy.ORCID 0000-0002-6974-9808
Robert L BradfordNorth Carolina Translational and Clinical Sciences Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States.ORCID 0000-0003-0908-1428
Gabriel A BratDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0003-3928-5931
Mario CannataroData Analytics Research Center, University Magna Graecia of Catanzaro, Catanzaro, Italy.ORCID 0000-0003-1502-2387
James J CiminoInformatics Institute, University of Alabama at Birmingham, Birmingham, AL, United States.ORCID 0000-0003-4101-1622
Noelia García-BarrioDepartment of Informatics, 12 de Octubre University Hospital, Madrid, Spain.ORCID 0000-0002-2789-8426
Nils GehlenborgDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0003-0327-8297
Marzyeh GhassemiDepartment of Computer Science and Medicine, University of Toronto, Toronto, ON, Canada.ORCID 0000-0001-6349-7251
Alba Gutiérrez-SacristánDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-1245-198X
David A HanauerDepartment of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, United States.ORCID 0000-0001-6931-3791
John H HolmesDepartment of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States.ORCID 0000-0003-2167-3602
Chuan HongDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0001-7056-9559
Jeffrey G KlannDepartment of Medicine, Harvard Medical School, Boston, MA, United States.ORCID 0000-0003-2043-1601
Ne Hooi Will LohNational University Health Systems, Singapore, Singapore.ORCID 0000-0002-4114-1286
Yuan LuoDepartment of Preventive Medicine, Northwestern University, Chicago, IL, United States.ORCID 0000-0003-0195-7456
Kenneth D MandlComputational Health Informatics Program, Boston Children's Hospital, Boston, MA, United States.ORCID 0000-0002-9781-0477
Mohamad DaniarClinical Research Informatics, Boston Children's Hospital, Boston, MA, United States.ORCID 0000-0001-9031-0835
Jason H MooreInstitute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, United States.ORCID 0000-0002-5015-1099
Shawn N MurphyDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-1905-8806
Antoine NeurazDepartment of Biomedical Informatics, Necker-Enfant Malades Hospital, Assistance Publique - Hôpitaux de Paris, Paris, France.ORCID 0000-0001-7142-6728
Kee Yuan NgiamNational University Health Systems, Singapore, Singapore.ORCID 0000-0001-5676-2520
Gilbert S OmennDepartment of Computational Medicine & Bioinformatics, University of Michigan, Ann Arbor, MI, United States.ORCID 0000-0002-8976-6074
Nathan PalmerDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-4361-207X
Lav P PatelDepartment of Internal Medicine, Division of Medical Informatics, University of Kansas Medical Center, Kansas City, KS, United States.ORCID 0000-0002-8626-137X
Miguel Pedrera-JiménezDepartment of Informatics, 12 de Octubre University Hospital, Madrid, Spain.ORCID 0000-0003-0187-3826
Piotr SlizComputational Health Informatics Program, Boston Children's Hospital, Boston, MA, United States.ORCID 0000-0002-6522-0835
Andrew M SouthSection of Nephrology, Department of Pediatrics, Brenner Children's Hospital, Wake Forest School of Medicine, Winston Salem, NC, United States.ORCID 0000-0002-3204-4142
Amelia Li Min TanDepartment of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0003-0623-6623
Deanne M TaylorDepartment of Biomedical and Health Informatics, The Children's Hospital of Philadelphia, Philadelphia, PA, United States.ORCID 0000-0002-3302-4610
Bradley W TaylorClinical and Translational Science Institute, Medical College of Wisconsin, Milwaukee, WI, United States.ORCID 0000-0002-6414-4172
Carlo TortiDepartment of Medical and Surgical Sciences, University Magna Graecia of Catanzaro, Catanzaro, Italy.ORCID 0000-0001-7631-5453
Andrew K VallejosClinical and Translational Science Institute, Medical College of Wisconsin, Milwaukee, WI, United States.ORCID 0000-0002-6543-5430
Kavishwar B WagholikarDepartment of Medicine, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-6219-861X
Consortium For Clinical Characterization Of COVID-19 By EHR (4CE)See Acknowledgments, .
Griffin M Weber *Department of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-2597-881X
Tianxi Cai *Department of Biomedical Informatics, Harvard Medical School, Boston, MA, United States.ORCID 0000-0002-5379-2502
Funding
Northwestern University Clinical and Translational Science Institute (NUCATS)UL1TR001422 · NCATS · NORTHWESTERN UNIVERSITY AT CHICAGO · PI D'AQUILA, RICHARD · 2015 to 2023
$56.8MRe-Entry Supplement: Investigation of Oral Microbial Enzymes for the Detection and Treatment of Periodontal DiseaseUL1TR002489 · NCATS · UNIV OF NORTH CAROLINA CHAPEL HILL · PI BUSE, JOHN BERNARD, SHAHEEN, NICHOLAS J · 2018 to 2022
$48.6MNorth Carolina Translational & Clinical Sciences Institute (NC TraCS)UL1TR001111 · NCATS · UNIV OF NORTH CAROLINA CHAPEL HILL · PI BUSE, JOHN BERNARD, CAREY, TIMOTHY S · 2013 to 2017
$43.9MWake Forest Clinical and Translational Science AwardUL1TR001420 · NCATS · WAKE FOREST UNIVERSITY HEALTH SCIENCES · PI ARD, JAMY D, FOLEY, KRISTIE L · 2015 to 2023
$32.3MLeveraging electronic health records to optimize treatment selection and response in multiple sclerosisR01NS098023 · NINDS · UNIVERSITY OF PITTSBURGH AT PITTSBURGH · PI Zongqi Xia · 2016 to 2026
$4.6MVisualizing healthcare system dynamics in biomedical Big DataU01CA198934 · NCI · HARVARD MEDICAL SCHOOL · PI WEBER, GRIFFIN M · 2015 to 2017
$1.9MBiases introduced by filtering electronic health records for patients with "complete data"R01LM013345 · NLM · HARVARD MEDICAL SCHOOL · PI WEBER, GRIFFIN M · 2020 to 2023
$1.4MPredoctoral Training Program in Biomedical Data Driven Discovery (BD3)T32LM012203 · NLM · NORTHWESTERN UNIVERSITY AT CHICAGO · PI KLABJAN, DIEGO, STARREN, JUSTIN B. · 2015 to 2019
$1.1MThe role of angiotensin-(1-7) in hypertension and hypertension-induced heart and kidney damageK23HL148394 · NHLBI · WAKE FOREST UNIVERSITY HEALTH SCIENCES · PI SOUTH, ANDREW MICHAEL · 2020 to 2024
$783kCharacterizing Population Differences between Clinical Trial and Real World PopulationsK99NS114850 · NINDS · HARVARD MEDICAL SCHOOL · PI BEAULIEU-JONES, BRETT K · 2021 to 2022
$186kBritish Heart Foundation FS/19/52/34563NCATS NIH HHS UL1 TR001111NCATS NIH HHS UL1 TR001420NCATS NIH HHS UL1 TR001422NCATS NIH HHS UL1 TR002489NCI NIH HHS U01 CA198934NHLBI NIH HHS K23 HL148394NHLBI NIH HHS L40 HL148910NINDS NIH HHS K99 NS114850NINDS NIH HHS R01 NS098023NLM NIH HHS R01 LM013345NLM NIH HHS T32 LM012203
Abstract
Coincident with the tsunami of COVID-19-related publications, there has been a surge of studies using real-world data, including those obtained from the electronic health record (EHR). Unfortunately, several of these high-profile publications were retracted because of concerns regarding the soundness and quality of the studies and the EHR data they purported to analyze. These retractions highlight that although a small community of EHR informatics experts can readily identify strengths and flaws in EHR-derived studies, many medical editorial teams and otherwise sophisticated medical readers lack the framework to fully critically appraise these studies. In addition, conventional statistical analyses cannot overcome the need for an understanding of the opportunities and limitations of EHR-derived studies. We distill here from the broader informatics literature six key considerations that are crucial for appraising studies utilizing EHR data: data completeness, data collection and handling (eg, transformation), data type (ie, codified, textual), robustness of methods against EHR variability (within and across institutions, countries, and time), transparency of data and analytic code, and the multidisciplinary approach. These considerations will inform researchers, clinicians, and other stakeholders as to the recommended best practices in reviewing manuscripts, grants, and other outputs from EHR-data derived studies, and thereby promote and foster rigor, quality, and reliability of this rapidly growing field.
Indexed as
Electronic Health RecordsCOVID-19Data CollectionHumansPeer Review, ResearchPublishingReproducibility of ResultsSARS-CoV-2COVID-19data qualityelectronic health recordsliteraturepublishingqualityreal-world datareporting checklistreporting standardsreviewstatistics
Identifiers
What OpenQuestion holds
Textmetadata
LicenceCC BY
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.