Lesson 17 of 25 · 7 min

What more runs buy, quantified, and what they do not

Four times the work halves the noise, and the noise was never the part of the answer that was wrong.

Track 4 lesson 12 said more runs reduce sampling noise and do nothing about the assumptions. Both halves of that statement have numbers attached, and the numbers are the point.

What a simulation reports is usually a percentile, and the precision of a sample percentile has a formula. It depends on the number of runs and on how densely the outcomes are packed at the level being measured.

standard error of a percentile = √(p × (1 − p) ÷ n) ÷ f(q)

p: The percentile expressed as a fraction. For a fifth percentile, 0.05.
n: The number of simulated runs.
f(q): How densely outcomes are packed around the level being estimated. Sparse packing means a noisier estimate.
q: The value of the percentile itself, in dollars.

Applied to the fifth percentile of the lesson 13 facility, the formula gives $23,117 at 1,000 runs, $7,310 at 10,000, and $2,312 at 100,000. Measuring it directly, by running each configuration eighty separate times and taking the spread of the answers, gives $22,549, $7,925 and $2,207. Ten times the runs buys 3.16 times the precision, because the error falls with the square root of the count. Quadruple the runs to halve the error; multiply by a hundred to gain one digit.

Now hold the runs at 10,000 and move the volatility input across its own two-standard-error band, 21.82 to 24.18 percent. The fifth percentile moves from $891,307 to $819,697, a spread of $71,610 — nine times the sampling error at that run count, and produced by an input whose uncertainty is not a modelling choice but a fact about how much data exists.

At 10,000 runs the answer is precise to about $7,900 and uncertain by about $71,600. Adding runs addresses the first figure and cannot touch the second.

Key terms

Sampling error: Variation in a result caused only by having drawn a finite number of paths. Falls with the square root of the run count and is the one error more computing fixes.
Parameter uncertainty: The range an input could reasonably take given the data available to estimate it. Independent of how many runs are performed.
Square-root scaling: The rule that precision improves with the square root of effort. Four times the runs for half the error, a hundred times for a tenth.

Worked example

Two sources of error, priced against each other

Fifth percentile of three-year ending collateral for the lesson 13 facility. Formula figures beside figures measured over eighty repetitions.

1,000 runs: $23,117 by formula, $22,549 measured
10,000 runs: $7,310 by formula, $7,925 measured
100,000 runs: $2,312 by formula, $2,207 measured
Precision gained from 100 times the runs: 10×
Volatility across two standard errors: $819,697 to $891,307
Ratio of the two uncertainties at 10,000 runs: About 9 to 1

A hundred-fold increase in computing closes $5,000 of a $71,600 gap, and the remaining $71,600 is not a computing problem.

Check yourself

A simulation at 10,000 runs reports a fifth percentile of $855,545 with a sampling error of about $7,900. Someone proposes running a million paths. What does that buy?

Saved in this browser only. No account, nothing sent anywhere.

Educational only. Nothing in this lesson is investment, legal or tax advice, or a recommendation to enter into any transaction. Figures are illustrative and computed from the inputs stated in the lesson.