Lesson 24 of 25 · 7 min
Backtesting, overfitting, and what validation cannot establish
Three years of daily data accepts a model that is fifty percent wrong about three times out of four.
Backtesting a risk model means counting how often losses exceeded the stated level and asking whether that count is consistent with the claim. It is a real test, and it is a weak one on any sample a practitioner actually has.
Three years of daily data is 756 observations. A correctly specified 99 percent model produces 756 × 0.01 = 7.56 expected exceptions, with a standard deviation of √(756 × 0.01 × 0.99) = 2.74. The range that a correct model produces 95 percent of the time runs from 3 to 13 exceptions, and any count inside it passes.
Now feed the test a model whose true exceedance rate is 1.5 percent rather than 1 percent — fifty percent wrong, in the direction that understates risk. Its expected count is 756 × 0.015 = 11.34, which sits comfortably inside the acceptance range. The probability it passes is 74.9 percent. A model that is twice as wrong, at 2 percent, still passes 35.0 percent of the time. A model that is too conservative, at 0.5 percent, passes 72.9 percent of the time.
Searching over specifications makes this worse in a way that is easy to quantify. Try twenty variants and keep whichever passes at a 5 percent level, and the chance that at least one passes by luck alone is 1 − 0.95²⁰ = 64.15 percent. The reported result is a test statistic; the unreported nineteen attempts are what determines whether it means anything.
A haircut policy validated over a decade containing no episode has been validated against a period containing no test. Passing is evidence of not being detectably wrong on this sample, which is a much smaller claim than being right.
Key terms
- Backtest
- Counting how often realised losses exceeded a model's stated level, and comparing that count with what the model implies.
- Acceptance region
- The range of exception counts a correct model produces most of the time. Wide on realistic samples, which is why weak models survive.
- Specification search
- Trying multiple model variants and reporting the one that passes. It raises the chance of a spurious pass from 5 percent to 64 percent across twenty attempts.
Worked example
What a three-year backtest can detect
756 daily observations, a 99 percent model, a 95 percent acceptance region of 3 to 13 exceptions.
- Expected exceptions if correct
- 7.56
- Model with a true rate of 1.5 percent
- Passes 74.9% of the time
- Model with a true rate of 2 percent
- Passes 35.0% of the time
- Model with a true rate of 0.5 percent
- Passes 72.9% of the time
- Chance one of twenty specifications passes by luck
- 64.15%
- What a pass establishes
- Not detectably wrong on 756 days
Standard deviation 2.74.
Expected count 11.34.
1 − 0.95²⁰.
A model understating risk by half survives this test three times in four, and a search across twenty variants finds a passing one by chance more often than not.
Check yourself
A 99 percent risk model is backtested over 756 days and produces 11 exceptions against an expected 7.56. It passes. What has been established?
Saved in this browser only. No account, nothing sent anywhere.
Educational only. Nothing in this lesson is investment, legal or tax advice, or a recommendation to enter into any transaction. Figures are illustrative and computed from the inputs stated in the lesson.