Lesson 23 of 25 · 7 min

Model risk: the error that is not in the data

Three errors sit inside every simulated number, and which one dominates depends on how far into the tail you look.

Three distinct errors sit inside any simulated figure, and they are routinely collapsed into a single vague reservation about models being imperfect. Separating them changes what you would do about each.

The three, in order of how easily they are removed:

Sampling error, from running a finite number of paths. Removed by computing, and quantified in lesson 17.
Parameter error, from estimating inputs on a finite sample. Removed only by more data, and the data arrives at one day per day.
Model error, from the shape assumed. Not removable by either, because no amount of data generated under the model's own assumptions can reveal that they are the wrong assumptions.

Decompose one quantity to see how they compare. Take the three-year ending collateral from the lesson 13 facility. At the fifth percentile the base figure is $855,545. Sampling standard deviation at 10,000 runs is about $7,900. The volatility input across two standard errors gives $819,697 to $891,307, a width of $71,610. And switching to a fat-tailed distribution at the same volatility gives $860,911 — a gap of $5,366, smaller than the sampling noise.

Now go further out. At the one-tenth percentile the base is $486,498, the parameter width is $63,789, and the model gap is $89,892. The ranking has inverted. In the body of the distribution the choice of distribution is almost irrelevant and the parameter estimate dominates; deep in the tail the choice of distribution is the answer and a better volatility estimate cannot touch it.

The figures a lender uses to set a routine haircut sit where parameter error dominates and better data helps. The figures that matter during a liquidation sit where model error dominates and better data does not exist.

Key terms

Model error: Error from having assumed the wrong form. Invisible to any test conducted inside the model, because such tests presuppose the form.
Parameter error: Error from estimating inputs on a finite sample. It has a standard error attached and shrinks, slowly, with more observations.
Error decomposition: Separating a figure's uncertainty into its sources so that effort can go to the one that dominates, which changes with how far into the tail the figure sits.

Worked example

One figure, three errors, two depths

Three-year ending collateral for the lesson 13 facility. Volatility 23 percent, band of two standard errors from three years of daily data.

Fifth percentile, base: $855,545
Sampling error at 10,000 runs: About $7,900
Parameter band at the fifth percentile: $819,697 to $891,307
Model gap at the fifth percentile: $5,366
One-tenth percentile, base: $486,498
Model gap at the one-tenth percentile: $89,892

The model choice contributed $5,366 at the fifth percentile and $89,892 at the one-tenth. The same three errors, ranked differently by depth.

Check yourself

At the fifth percentile, changing from a normal to a fat-tailed distribution moves the answer by $5,366 while the volatility estimate's own uncertainty moves it by $71,610. At the one-tenth percentile the model gap is $89,892. What follows?

Saved in this browser only. No account, nothing sent anywhere.

Educational only. Nothing in this lesson is investment, legal or tax advice, or a recommendation to enter into any transaction. Figures are illustrative and computed from the inputs stated in the lesson.