What is an ordinal Brier score?

A standard Brier score treats every wrong answer as equally wrong. On a question whose answers have an inherent order, that is the wrong judgment: being one place out is not the same failure as being three places out. The ordinal Brier score gives partial credit for near misses. This page assumes you have read Brier scores.

When ordering matters

"Will Country X hold national elections before July 1?" has two answers and no ordering. "How many seats will Party A win in the next election?" is different. Its answers are ranges laid out along a number line:

Party A wins 80 seats, so "76 to 100" is correct.

• 45% on "50 to 75". This missed, but it missed by one bucket.
• 15% on "More than 100". This also missed by one bucket, in the other direction.
• 15% on "Fewer than 50". This was further from the correct answer, and therefore "more" wrong.

A scoring rule that charges the same for all three is discarding information the forecaster actually captured.

What a standard Brier score does here

Scored the ordinary way, each answer is compared against 1 if it occurred and 0 if it did not, and the squared errors are summed:

0.81 out of a possible 2.0. The rule has noticed that most of the probability was not on the winning answer, and nothing else.

The cutpoint method

The ordinal score fixes this by scoring the question as a series of simpler binary questions. Instead of asking "which answer?", it asks "is the answer above or below this line?" at every line that can be drawn between two answers. Four answers give three lines:

At each cutpoint, add up the probability the forecaster placed below the line, add up the probability placed above it, and score those two totals as a two-answer Brier score against where the true answer fell:

cutpoint score = (mass below - outcome below)² + (mass above - outcome above)²
ordinal daily score = mean of the cutpoint scores

This is what converts distance into cost. Probability placed just on the wrong side of one line is on the right side of the other two, so it is charged once instead of three times. Probability placed at the far end of the scale is on the wrong side of every line.

Working the example

The true answer is 80 seats, so it sits above cutpoints 1 and 2, and below cutpoint 3.

0.27 against 0.81. The same forecast, on the same outcome, charged a third as much, because the rule now credits the forecaster for having put 70% of their probability within one bucket of the truth. Cutpoint 2 carries the remaining cost, which is the right place for it: that is the one line the forecaster misjudged, having placed 60% below a line the answer sat above.

Scale and the two-answer case

The scale does not change. An ordinal Brier score still runs from 0.0 to 2.0. Placing everything on the answer at the far end of the scale from the truth scores 2.0, exactly as placing everything on the wrong answer of a two-answer question does. This is what lets ordinal and standard scores share a leaderboard column without rescaling.

On a two-answer question it is the standard score. Two answers give exactly one cutpoint, and that cutpoint's score is the ordinary Brier score. The ordinal rule is a generalization rather than an alternative, which is why turning it on cannot quietly change what a binary question was worth.

When not to use it

Ordinal scoring is only correct when the answers really are ordered, and the order really is the one they are listed in. "Fewer than 50 / 50 to 75 / 76 to 100 / More than 100" qualifies. "Which party wins the most seats?" does not: there is no sense in which one party is between two others, so partial credit for an adjacent answer would be credit for nothing.

Applying it to an unordered set breaks the score. It hands out partial credit based on the order the options happened to be written in, which means a forecaster can improve their score by reasoning about the option list rather than about the world. Use it for ranges, dates, and counts. Use the standard rule for everything else.

A third variant: independent options

There is one more case neither rule covers. Hinsley also supports questions where several outcomes may be correct, or none. "Which of these five countries will hold an election before June?" is not a choice between mutually exclusive answers, so its probabilities do not sum to 1 and the answers do not compete.

Hinsley scores these as a set of independent two-answer questions, one per option, and takes the mean. Each option contributes exactly what it would have scored as a standalone question, so the range is again 0.0 to 2.0 and a single-option question scores identically to the equivalent binary one.

multi-outcome daily score = mean over options of 2 × (forecast - outcome)²

One consequence matters in practice: because every option is scored, a forecaster has to state a probability for every option in play. An option left blank cannot be read as 0% without turning silence into a confident claim, so Hinsley declines to score the day rather than guess.

How Hinsley calculates it

Ordinal scoring is a per-question setting, chosen when the question is authored. It applies to the whole question for its whole life.

• The answer order is the cutpoint order. Cutpoints are derived by walking the answer list in the order it is defined, so an ordinal question's answers must be listed from lowest to highest. This is the single thing to get right when authoring one.
• Daily first, then averaged. The ordinal rule replaces the daily calculation only. Everything above it is unchanged: one score per day of the scoring window, averaged into a per-question mean, then compared against the baseline to give a relative Brier score.
• One column on the leaderboard. Standard, ordinal, and multi-outcome scores share a column, because a question is scored under exactly one of the three and all three are on the same 0-to-2 scale. A leaderboard mixing all three is still a valid ranking.

This guide was written with the help of AI, but was reviewed and edited by a human.