Evidence map›Paper›PMID 35022756›Full record

ArticleJournal of the American Medical Informatics Association : JAMIA2022

Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees.

Jean Feng, Alexej Gossmann, Berkman Sahiner, Romain Pirracchio

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Journal of the American Medical Informatics Association : JAMIA, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 7 papers.

0numbers the graph read from it
0cells of the map it votes in
7citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

7 citing papers in PubMed.

  1. Article
  2. Article
  3. Article
  4. Review
  5. Article
  6. Review
  7. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

4 authors.

Jean FengDepartment of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California, USA.
Alexej GossmannCDRH-Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, Maryland, USA.
Berkman SahinerCDRH-Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, Maryland, USA.
Romain PirracchioDepartment of Anesthesia and Perioperative Care, University of California, San Francisco, San Francisco, California, USA.

Funding

UCSF-Stanford Center of Excellence in Regulatory Science and InnovationU01FD005978 · FDA · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI ALTMAN, RUSS BIAGIO, GIACOMINI, KATHLEEN M · 2016 to 2025
$34.5M
Center of Excellence in Regulatory Science and Innovation grant to University of CaliforniaFDA HHS U01 FD005978Food and Drug Administration (FDA) of the U.S. Department of Health and Human Services (HHS)San Francisco (UCSF)Stanford University U01FD005978University of California, San Francisco (UCSF) and Stanford University U01FD005978
6 · The paper itself

Abstract

objectiveAfter deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees. MATERIALS AND

methodsWe introduce 2 procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting Chronic Obstructive Pulmonary Disease (COPD) risk. We derive "Type I and II" regret bounds, which guarantee the procedures are noninferior to a static model and competitive with an oracle logistic reviser in terms of the average loss.

resultsBoth procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI, 0.818-0.938). Online recalibration using BLR and MarBLR improved the aECI towards the ideal value of zero, attaining 0.265 (95%CI, 0.230-0.300) and 0.241 (95%CI, 0.216-0.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average area under the receiver-operating characteristic curve (aAUC) from 0.767 (95%CI, 0.765-0.769) to 0.800 (95%CI, 0.798-0.802) and 0.799 (95%CI, 0.797-0.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI, 0.913-0.935) and 0.925 (95%CI, 0.914-0.935), compared to the static model's aAUC of 0.904 (95%CI, 0.892-0.916). DISCUSSION: Despite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data.

conclusionsBLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time.

Indexed as

Models, StatisticalPulmonary Disease, Chronic ObstructiveBayes TheoremHumansLogistic ModelsPrognosisBayesian model updatingclinical prediction modelsmachine learningmodel recalibration

Identifiers

PMID35022756
PMCPMC9006691

What OpenQuestion holds

Textmetadata
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.