What is calibration?

The three Brier measures ask how much error a forecaster produced. Calibration asks something different and, for a decision-maker, often more useful: when this forecaster says 70%, does the thing happen 70% of the time?

What calibration means

Calibration is a property of a body of work, not of a single forecast. Take every occasion on which a forecaster said "about 70%", and check how many of those things actually happened. If the answer is close to 70%, they are well calibrated at 70%. Repeat across the range and you have their calibration.

This is the property that makes a probability usable. If a well-calibrated forecaster tells you 70%, you can act on 70%: plan for the other 30%, price the risk at 30%, and see the other outcome three times in ten without anything having gone wrong. If a poorly calibrated forecaster tells you 70%, you have to work out what they mean by it before you can use it, and the number has stopped carrying information on its own.

Being uncertain when uncertainty is warranted is as much a skill as accuracy. Calibration treats that as a virtue.

The calibration curve

A calibration curve is built by sorting every forecast into bins by the probability stated, then comparing two numbers in each bin: the average probability stated, and the share of those cases that came true. Perfect calibration puts every bin on the diagonal, where the two are equal.

Here is a forecaster with a common and instructive shape:

Read the gap column from top to bottom and the diagnosis is unambiguous. When this forecaster says something has a 95% chance of happening, those events only occur 83% of the time. On the lower end, when they say there is a 5% chance, those things happen 10% of the time. They are pushing their probabilities too far toward the extremes in both directions. This forecaster exhibits overconfidence.

The opposite shape - bins above the diagonal at the top and below it at the bottom - is underconfidence.

Calibration is not accuracy

Calibration on its own can be achieved by a forecaster who knows nothing at all. Consider someone who answers every question with the historical base rate for that class of question - say 12% for "will there be a coup this year" - and never moves. Over a large sample, roughly 12% of those will happen. Their curve sits on the diagonal. They are perfectly calibrated and completely uninformative.

The missing property is discrimination: the ability to separate the cases that will happen from the cases that will not, by saying different things about them. A forecaster with good discrimination gives high probabilities to things that occur and low probabilities to things that do not, and so their bins are not all clustered near the base rate.

Good forecasting needs both, and they are independent. A forecaster can have either without the other, which is exactly why a leaderboard position and a calibration curve are shown side by side rather than one summarizing the other.

Splitting a Brier score into three terms

Allan Murphy showed in 1973 that a Brier score splits into three terms.

Brier = reliability - resolution + uncertainty

The third term is the useful one for interpreting a leaderboard. Uncertainty measures nothing about the forecaster's skill: it is the base rate multiplied by one minus the base rate, where the base rate is the share of the scored observations that came true. Two forecasters who were scored on the same questions over the same days therefore carry the same uncertainty. It is the arithmetic behind the warning that absolute Brier scores are not comparable between forecasters who answered different questions.

The first term is the useful one for improving. Reliability is the only one of the three a forecaster can move by changing how confident they sound without learning anything new about the subject. Resolution requires actually knowing more.

Two summary numbers

A curve is the honest presentation, but a curve cannot be sorted or tracked over time. Two summary statistics are usually quoted beside it.

Expected calibration error, or ECE, is the average distance between the curve and the diagonal, weighted by how many forecasts each bin holds. It is easy to explain and easy to misread. ECE is biased upward on small samples - a genuinely well-calibrated forecaster with twenty forecasts will still post a non-zero one, purely from sampling noise - and it moves when the number of bins changes, so two ECE figures are only comparable when both were computed the same way.

Cox calibration avoids both problems by fitting a line rather than counting bins. It regresses the outcomes on the stated probabilities and reports two numbers:

• Slope. 1.0 is perfect. Below 1.0 means overconfident: probabilities pushed further from 50% than the evidence supported. Above 1.0 means underconfident: probabilities held closer to 50% than the evidence supported.
• Intercept. 0.0 is perfect. It is the bias left over once that stretch is accounted for - a forecaster who is too high or too low across the whole range rather than too confident.

The forecaster in the table above would return a slope below 1.0 and an intercept near zero: confident by the wrong amount.

How to act on a calibration curve

Calibration is the most improvable part of forecasting, because the fix does not require knowing more about the subject.

• If the curve sags below the diagonal at the top and rises above it at the bottom, you are overconfident. Move your probabilities toward 50%. In practice this means treating 95% as the number you reserve for cases that are effectively settled, rather than for cases you feel sure about.
• If the curve bulges the other way, you are underconfident, and you should be committing further. This is the only case in which pushing probabilities toward the extremes will improve your scores.
• If the whole curve sits above or below the diagonal, you have a directional bias rather than a confidence problem. Look for whether you are consistently optimistic or pessimistic about a class of subject.
• If the bins are all clustered near the base rate, your calibration may be fine and your discrimination is the problem. That one is not fixed by adjusting confidence, only by better research.

One warning about all four: read them off the curve, not off a single number, and check the sample size in each bin before you act. A bin holding six forecasts says almost nothing.

How Hinsley calculates it

Hinsley builds a calibration curve from every scored forecast a forecaster has made, using ten bins by default. Each scored day of each resolved question contributes an observation, and a question with several answers contributes one observation per answer.

• Two weightings, answering two questions. By day, every observation counts once, so a forecast that stood for two hundred days counts two hundred times. That answers "can you trust the numbers this forecaster puts on screen", because the claim was made on each of those days. By question, each question's weight is divided among its observations so every question counts once. That answers "who is better calibrated", where a long-running question is not two hundred times the evidence. Neither is more correct, and Hinsley shows both.
• The fit is withheld rather than guessed. Hinsley reports no slope or intercept below twenty observations, and none when the data cannot support a fit - no spread in the probabilities, an outcome that never varies, or a forecaster whose forecasts separate the outcomes perfectly. An unidentified fit is shown as absent, not as a number.
• The decomposition uses its own Brier score. The Brier figure quoted beside the decomposition is the mean squared error over the observations in the curve, one answer at a time. It is not the mean Brier score on the scorecard, which sums across a question's whole answer set. The two are on different scales by design and should not be compared.

That completes the four measures. Absolute error tells you how far off a forecaster was, the relative score tells you whether they beat the crowd and when, ordinal scoring handles answers that sit on a scale, and calibration tells you whether the probabilities can be read literally. A forecasting program needs all four, because each one can be strong while another is not.

This guide was written with the help of AI, but was reviewed and edited by a human.