Evidence map›Paper›PMID 34340529›Full record

ReviewCirculation. Cardiovascular quality and outcomes2021

External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature.

Benjamin S Wessler, Jason Nelson, Jinny G Park, Hannah McGinnes, Gaurav Gulati, Riley Brazil, Ben Van Calster, David van Klaveren, Esmee Venema, Ewout Steyerberg and 2 more

Abstract readReview
PubMed Full text at PMC Publisher
In one paragraph

Review in Circulation. Cardiovascular quality and outcomes, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 45 papers, 5 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
45citing papers in PubMed, 5 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

45 citing papers in PubMed, 5 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Guideline
  4. Pooled it
  5. Pooled it
  6. Article
  7. Review
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Review
  14. Review
  15. Review
  16. Article
  17. Review
  18. Article
  19. Review
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

12 authors.

Benjamin S WesslerPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
Jason NelsonPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
Jinny G ParkPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
Hannah McGinnesPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
Gaurav GulatiPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
Riley BrazilPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
Ben Van CalsterKU Leuven, Department of Development and Regeneration, Belgium (B.V.C.).
David van KlaverenPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
Esmee VenemaDepartment of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands.
Ewout SteyerbergDepartment of Biomedical Data Sciences (E.S.), Leiden University Medical Centre, Netherlands.
Jessica K PaulusPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
David M KentPredictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.

Funding

Palliative Care for High-Risk TAVR Patients: The Impact of MultimorbidityK23AG055667 · NIA · TUFTS MEDICAL CENTER · PI WESSLER, BENJAMIN SETH · 2018 to 2022
$948k
Improving Care for Patients with Aortic Stenosis and MultimorbidityR03AG056447 · NIA · TUFTS MEDICAL CENTER · PI WESSLER, BENJAMIN SETH · 2017 to 2018
$263k
NIA NIH HHS K23 AG055667NIA NIH HHS R03 AG056447
6 · The paper itself

Abstract

backgroundThere are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated.

methodsA SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data.

resultsWe identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717;

conclusionsMany published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

Indexed as

Cardiovascular DiseasesHumansROC Curvecalibrationcardiovascular diseasedecision makingliterature review

Identifiers

PMID34340529
PMCPMC8366535

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC-ND
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.