Indication of Measures of Uncertainty for Statistical Significance in Abstracts of Published Oncology Trials: A Systematic Review and Meta-analysis.

importanceThere is growing consensus that reliance on P values, particularly a cutoff level of .05 for statistical significance, is a factor in the challenges in scientific reproducibility. Despite this consensus, publications describing clinical trial results with P values near .05 anecdotally use declarative statements that do not express uncertainty.

objectivesTo quantify uncertainty expression in abstracts describing the results of cancer randomized clinical trials (RCTs) with P values between .01 and .10 and examine whether trial features are associated with uncertainty expression. DATA SOURCES: A total of 5777 prospective trials indexed on HemOnc.org, as of September 15, 2019. STUDY SELECTION: Two-arm RCTs with a superiority end point with P values between .01 and .10. DATA EXTRACTION AND SYNTHESIS: Abstracts were evaluated based on an uncertainty expression algorithm. Ordinal logistic regression modeling with multiple imputation was performed to identify whether characteristics of study design, results, trial authors, and context P values were normalized by dividing by prespecified α value. MAIN OUTCOMES AND MEASURES: Uncertainty expression in abstracts as determined by the algorithm and its association with trial and publication characteristics.

resultsOf 5777 trials screened, 556 met analysis criteria. Of these, 222 trials (39.9%) did not express uncertainty, 161 trials (29.0%) expressed some uncertainty, and 173 trials (31.1%) expressed full uncertainty. In ordinal logistic regression with multiple imputation, trial features with statistically significant associations with uncertainty expression included later year of publication (odds ratio [OR], 1.70; 95% CI, 1.24-2.32; P < .001), normalized P value (OR, 1.36; 95% CI, 1.11-1.67; P = .003), noncooperative group studies (OR, 1.72; 95% CI, 1.12-2.63; P = .01), and reporting an end point other than overall survival (OR, 1.41; 95% CI, 1.01-1.96; P = .047). Funding source, number of authors, journal impact tier, author nationality, study of unapproved drugs, abstract word count, whether the marginal end point was a primary or coprimary end point, and effect size (in subgroup analysis) did not have statistically significant associations with uncertainty expression. CONCLUSIONS AND RELEVANCE: Published oncology articles with marginally significant results may often incompletely convey uncertainty. Although it appears that more uncertainty is expressed in recent abstracts, full uncertainty expression remains uncommon, and seemingly is less common when reporting overall survival, results with P values lower than α levels, and cooperative group studies.