What is a relative Brier score?

In Hinsley, a relative Brier score measures a forecaster against the crowd. It is the number Hinsley ranks leaderboards on, because, unlike a standard Brier score, it stays meaningful when forecasters choose their own questions and their own moment to answer. This page assumes you have read Brier scores.

The problem it solves

A mean Brier score is a fair comparison only when everyone answered the same questions on the same days. In a forecasting program with humans participating, this is never the case. Participants pick the subjects they know, skip the ones they do not, and enter questions at different moments.

That last freedom is the damaging one. Because scoring runs per day, a forecaster who waits until a question is nearly settled and then states the near-certain answer collects a small number of near-perfect days. Their mean Brier score can be the best on the question, while contributing nothing anyone could have acted on. Meanwhile the forecaster who was accurate a month early, when the call was genuinely hard, carries the higher-error days that early accuracy inevitably produces.

We want a scoring system that rewards forecasters for being accurate but also early.

Three forecasters, one question

Take the question "Will Country X hold national elections before July 1?" over a seven-day scoring window, and the elections are held. Three forecasters take part. Each row shows the probability given to yes, and the daily Brier score it earned.

Forecaster 1 is confident and early, and updates once on day 4.

Forecaster 2 is inaccurate throughout, and grows more confident in the wrong direction.

Forecaster 3 says nothing until day 6, then states what is by then nearly settled.

On mean Brier score, Forecaster 3 wins by a wide margin: 0.0002 against 0.0114. An absolute score cannot be used as a ranking once results like that are possible.

The baseline

The fix is to stop asking how much error a forecaster produced and start asking how much less error they produced than the crowd did, on each day separately. That requires a reference forecast, and the reference Hinsley uses is the unweighted mean of every live forecast on the question that day.

Averaging the live forecasts above, and scoring that average like any other forecaster:

On day 6, Forecaster 3 arrives with a strong forecast, the mean moves from 0.575 to 0.7133, and the baseline's own daily score improves from 0.3613 to 0.1644. The bar rises for everyone at the moment the crowd gets better. Beating the crowd late, when the question is often easier to forecast, naturally becomes more difficult. So forecasters are not rewarded for waiting until late in the question's lifecycle.

Calculating the relative score

Four steps:

1. Score the baseline forecast on every day of the scoring window.
2. For each day the forecaster had a live forecast, subtract their daily score from the baseline's daily score.
3. Sum those daily differences.
4. Divide the sum by the number of days in the window, not by the number of days the forecaster took part.

daily relative = baseline daily score - forecaster daily score
relative Brier score = sum of daily relatives / days in the window

The subtraction is that way round deliberately, and it makes higher better - the opposite of an absolute Brier score. A positive relative Brier score means the forecaster beat the crowd average. Zero means they matched it. Negative means the crowd average would have served you better.

Reading the result

The ranking has inverted compared to a mean Brier score:

Forecaster 3 still finishes above the crowd average, because their two days really were better than the crowd's. They are simply no longer credited with the five days on which they said nothing. Forecaster 1 has the best score by a factor of six, on a question where their absolute error was fifty times larger.

Why the denominator is every day

Astute readers will notice that step 4 is where we account for late entries. Dividing by the number of days the forecaster participated would hand the late entrant an average of their two best days and discard the rest of the question. Dividing by the length of the window treats every unanswered day as a day on which they added nothing (positive or negative) to the crowd - which is exactly what it was.

The consequence is that a relative Brier score behaves like a contribution. Being present and merely average earns nothing, because the daily difference is zero. Being absent earns nothing either. The only way to accumulate a positive score is to be better than the crowd on days when it was possible to be.

How Hinsley calculates it

Hinsley computes the baseline first, as its own scored forecaster on the question, then scores everyone else against the daily values it produced.

• Higher is better, and the sign is meaningful. A positive relative Brier score beat the crowd average. This is the reverse of the mean Brier column beside it, which is the most common source of confusion when reading a scorecard.
• The leaderboard totals rather than averages. Each resolved question contributes one relative Brier score, and a forecaster's leaderboard position is the sum across every question they were scored on, so breadth of participation counts. A per-question average is shown beside it for readers who want skill independent of volume.
• LLM forecasters rank alongside people. Every LLM that forecasts a question is scored against the same plain-average baseline as every human forecaster, so one board shows whether a given model beats the crowd, and where it places against the people in it.

A relative Brier score says a forecaster beat the crowd. It does not say their stated probabilities can be read literally - a forecaster can rank well while being systematically overconfident. That is what calibration measures.

This guide was written with the help of AI, but was reviewed and edited by a human.