'Optimism bias' in contemporary national clinical trial network phase III trials: are we improving?

Background: Previous studies have found that overestimating treatment effects (i.e. 'optimism bias') leads to underpowered clinical trials. The prevalence of 'optimism bias' in contemporary National Clinical Trials Network (NCTN) cancer clinical trials is unknown. Methods: We conducted a systematic review of NCTN phase III randomized trials published from January 2007 to January 2017. We compared the hypothesized versus observed treatment effects in each trial, and examined whether trial-related factors were correlated with the study results. We also reviewed the methods of each protocol to assess whether a rationale for the hypothesized effect size was provided. Results: We identified 161 clinical trials, of which 130 were eligible for analysis. Original protocols could not be located for 8 trials (5.0%). Twenty-eight trials (21.5%) observed a statistically significant difference in the primary end point favoring the experimental treatment. The median ratio of observed-to-expected hazard ratios among trials that observed a statistically significant effect on the primary end point was 1.07 (range: 0.33-1.28) versus 1.32 (range: 0.86-2.02) for trials that did not, compared with 1.34 and 1.86, respectively, for National Cancer Institute (NCI) trials published between 1955 and 2006. An effect size at least as large as the one projected in the protocol trials was observed in 9.8% of trials, compared with 17% of NCI trials published from 1955 to 2006. Most trials (64.6%) provided no rationale to support the magnitude of the proposed treatment effect in the protocol. Conclusions: Despite a reduction in 'optimism bias' compared with previous eras, most contemporary NCTN phase III trials failed to establish statistically significant benefits of new cancer therapies. Better rationalization of proposed effect sizes in research protocols is needed.