What is a Brier score?

A Brier score is the standard way to grade a probabilistic forecast. It was introduced by Glenn Brier in 1950 for weather forecasting, and it remains the default scoring rule in forecasting tournaments, as well as in Hinsley. It works by treating a probability as a claim about the world and measuring how far that claim sat from what happened.

The appeal of the rule is that it cannot be gamed. A Brier score is what statisticians call a proper scoring rule: the way to get the best expected score is to state the probability you actually believe. Rounding a 60% belief up to 90% to look decisive raises your expected error, and so does hedging a 90% belief down to 60% to look cautious.

What a Brier score measures

Suppose you are forecasting the question "Will Country X hold national elections before July 1?" and you forecast 90%. If the elections are held, you were mostly accurate, but not entirely: you also said, implicitly, that there was a 10% chance it would not. Both halves of that claim get scored.

A Brier score quantifies this as squared error. For each possible answer, take the probability you gave it, subtract what actually happened - 1 for the answer that occurred, 0 for every answer that did not - and square the difference. Squaring is what makes the rule proper, and it is why a confident miss costs much more than a hedged one.

Calculating a daily score

Your forecast of 90% yes, and therefore 10% no, against an outcome where the elections were held:

(0.9 - 1.0)² = 0.01
(0.1 - 0.0)² = 0.01
daily score = 0.01 + 0.01 = 0.02

The score for a day is the sum over every answer, not the average, so a two-answer question runs from 0.0 to 2.0. Watch for this when you compare against another source. The other common convention scores only the answer that occurred, and runs from 0.0 to 1.0 - exactly half of these numbers. Hinsley sums over the full answer set, so every score on this page is on the 0-to-2 scale.

Averaging over the days

A forecast is not a single moment. It stands from the day you submit it until you replace it, and it is scored on every one of those days. This is why you will often see a Brier score described as a mean squared error: the daily scores are averaged to produce the score for the question.

Say you made that 90% forecast a week before the deadline, and never changed it:

Every day is identical, so the mean is 0.02.

What happens when you update

Now suppose the electoral commission publishes a polling date on day 3, and you raise your forecast to 95% on day 4. The daily score for a 95% forecast is:

(0.95 - 1.0)² + (0.05 - 0.0)² = 0.0025 + 0.0025 = 0.005

(0.02 + 0.02 + 0.02 + 0.005 + 0.005 + 0.005 + 0.005) / 7 = 0.0114

Lower is better (less error), so the update improved the score from 0.02 to 0.0114. A forecaster who revisits a question as evidence arrives accumulates fewer high-error days than one who submits once and walks away, even when both end up at the same final probability.

Reading the number

Some reference points on a two-answer question, all on the 0-to-2 scale:

Remember the 0.50 row. Saying 50% on a two-answer question always scores exactly 0.50, whatever happens, so it is the line between adding information and destroying it. A mean Brier score above 0.50 across a body of work means a forecaster would have done better by refusing to commit.

What a Brier score cannot tell you

A mean Brier score measures error in absolute terms, which is what limits it. Three things it conflates:

Question difficulty. A forecaster who only participates in easy questions will post a better score than one who answers more difficult ones, and neither number says which of them is the better forecaster. Absolute scores are only comparable between people who answered the same questions on the same days.

Participation. Because scoring runs per day, a forecaster who enters a question on the final day, when the answer is nearly settled, collects a handful of near-perfect days and no others. Their mean Brier score can beat a forecaster who was accurate for a month. This is the problem the relative Brier score exists to solve.

Confidence honesty. A Brier score mixes together two different virtues: knowing which way a question will go, and stating the right level of confidence about it. A forecaster can score respectably while being systematically overconfident. Separating those two is what calibration is for.

How Hinsley calculates it

Hinsley scores a question once it resolves, over a scoring window that runs from the question opening to its resolution. For every day in that window it takes the forecast that stood on that day, sums the squared error across the question's full answer set, and stores the result as that day's score. Those daily scores are then averaged to give the forecaster's mean Brier score for the question, and the average runs only over the days the forecaster actually had a live forecast.

Three things to note about that design:

• Forecasts carry forward. You are scored on your standing forecast every day until you replace it.
• Days before your first forecast are not scored. They are excluded from your mean rather than counted as errors, which is why the mean Brier score alone cannot penalize late entry.
• People and models are scored identically. An AI forecaster, a crowd aggregation, and an individual analyst all produce a probability per day, so they all land on the same scale and the same leaderboard.

Questions whose answers have an inherent ordering use a variant of this calculation that gives partial credit for near misses. That variant is covered in ordinal Brier scores.

This guide was written with the help of AI, but was reviewed and edited by a human.