Observational studyJournal of the American Medical Informatics Association : JAMIA2023
Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study.
Observational study in Journal of the American Medical Informatics Association : JAMIA, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 9 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
9 citing papers in PubMed, 11 citations in OpenAlex.
- From study design to executable code: automating target trial emulation with large language models.JAMIA open · 2026Article
- A Standardized Guideline for Assessing Extracted Electronic Health Records Cohorts: A Scoping Review.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2025Article
- Advancing Medical Imaging Research Through Standardization: The Path to Rapid Development, Rigorous Validation, and Robust Reproducibility.Investigative radiology · 2025Review
- CohortDiagnostics: Phenotype evaluation across a network of observational data sources using population-level characterization.PloS one · 2025Article
- Clinical Research Informatics: a Decade-in-Review.Yearbook of medical informatics · 2024Review
- Using routinely collected clinical data for circadian medicine: A review of opportunities and challenges.PLOS digital health · 2024Review
- Why do probabilistic clinical models fail to transport between sites.NPJ digital medicine · 2024Review
- Variation in Phenotype Definitions in Observational Clinical Research: A Review of Three Conditions.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2024Review
- Development and evaluation of an algorithm to link mothers and infants in two US commercial healthcare claims databases for pharmacoepidemiology research.BMC medical research methodology · 2023Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
38 authors at 20 institutions in 9 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectiveObservational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics. MATERIALS AND
methodsNine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics.
resultsOn average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5.
conclusionsIndependent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.