What “calibrated” means — and how to read it on our pages

A favourite priced at 75¢ claims to win about three times in four. On Polymarket's crypto Up or Down markets, one minute before the close, 5-minute favourites priced 70–80¢ (75.1¢ on average) won 76.4% of 5,458 windows — 95% interval 75.3%–77.5%. The price is just below that interval: in this band the favourite won 1.3 pts more often than its price, a gap the sample is large enough to see.

In the same band, the two halves of the period gave 78.0% (Aug 7 – Sep 13) and 74.5% (Sep 14 – Oct 21) on 5-minute windows. At the bottom of the scale, 50–60¢ favourites cost 53.9¢ on average and won 56.0% of 5,030.

Up won 50.02% of the 123,946 windows we scored. A forecaster who always says 50% is therefore right on average — calibrated in its single band — and scores a Brier score of 0.25, the coin-flip mark. Calibration says the claims are honest; it does not say they are useful.

The order book halfway through a window is a real example. Its price was calibrated in 12 of 20 bands, against 14 of 20 one minute before the close. But its Brier score was 0.1464 against 0.0521: 2.8 times worse, because halfway through it says "maybe" far more often. Sharpness — how often a forecast commits to something near 0 or 1 — is the other half of the story.

Every band is a sample. With 5,458 5-minute windows in the 70–80¢ band, the 95% interval is ±1.1 pts around the observed rate. The same band on hourly windows has 202 windows and an interval of ±5.8 pts: an hourly band can miss its price by several points and still be consistent with a calibrated market.

The smallest band in the table above is hourly 60–70¢ with 172 windows; its interval is 61.9%–75.6%. A rule of thumb near 50%: the interval's half-width is about 1/√n, so 100 windows give ±10 points and 10,000 give ±1.

For each window, take the forecast probability of Up, subtract the outcome (1 if Up, 0 if Down) and square it; the Brier score is the mean over windows. 0 is perfect, 0.25 is a coin flip. One minute before the close the order book scored 0.0599 on 5-minute windows, 0.0345 on 15-minute ones and 0.0279 on hourly ones. Pooled over windows, a score weights each window equally; the nightly scores on /performance weight each night equally, so the two do not average to the same number.

• Each market page has a calibration table: the favourite's price one minute before the close, by 10¢ band, against how often it won, plus a one-line summary of whether outcomes beat prices.
• The performance page scores the market and our model every night with the Brier score, so you can see the variability from one night to the next.
• Our calibration study goes one level finer: 20 bands of 5 points with their 95% intervals, for the order book and for both of our models.