A model that says "70%" should be right about 7 times in 10 of those. That is what we test, and hit ratei alone does not test it.
Brier score
The Brieri score is the mean squared error of the probabilities. For one event it is (probability − outcome)², with outcome 1 if it happened and 0 if not. Lower is better.
A 50/50 forecast always scores 0.25. So a model is only worth something if, over many matches, it does better than the market on the same matches. For 1X2 there are three outcomes and the sum is taken across the three.
Calibration table
Group the forecasts by probability range and see how often the event happened in each group. In a well-calibrated model the two columns match.
| Forecast | Bets | Happened | Gap |
|---|---|---|---|
| 50% to 60% | 200 | 55% | small |
| 60% to 70% | 180 | 66% | small |
| 70% to 80% | 120 | 64% | overestimate |
Two models finding the same number of correct results does not make them equal. One may have reliable probabilities and the other overestimate. Calibration counts, not only success.
Practical rules
- Always compare with the market on the same matches, not with 0.25.
- You need hundreds of forecasts before you conclude anything.
- A model that overrates favourites loses money even if it "predicts" most results.