Scored in public, and impossible to edit.

How forecasts are scored#

• Accuracy of each probability. A forecast of 80% that a trial meets its goal scores well if it does and badly if it doesn't, in proportion to how confident it was (the Brier score).
• Honesty of the probabilities. Of all the trials we give 70%, about 70% should succeed. We show this as a calibration chart.
• Skill, not luck. Every score is compared with a simple baseline: how often trials of the same phase, disease area and design succeed. The model has skill only if it beats that baseline by more than chance. We report the margin with its uncertainty.
• Size of the effect (coming next). For trials with standard measures, we will also forecast how big the effect will be, and check whether the result falls inside our stated range. Today's forecasts cover whether each trial meets its main goal.

Why the record can't be edited#

1. Every night, the day's new forecasts are written to a file that is only ever added to.
2. A fingerprint of that file (a SHA-256 digest) is recorded the same day in a time-stamped history held outside our own servers. Changing even one character of any forecast would change the fingerprint.
3. When a trial reads out, its forecast is revealed along with the file it came from, so anyone can recompute the fingerprint and confirm the forecast was made before the result.
4. A forecast is never edited or deleted. If we change our mind before a readout, that is a new forecast with its own date; both are scored.

What we will never do#

• Count a backtest (a forecast of a trial whose result was already known) as a live forecast. Backtests are labelled and shown separately.
• Drop misses, or score only the trials we are confident about.
• Count a trial with no posted result as a failure. It stays unresolved.

Definitions and rules in full: Methodology.