QQ 301, 29 min read

Validation, check by check

A single backtest is one sample, at one parameter setting, with whatever costs were typed in. Validation reruns and resamples it to say how much was luck.

4.1 How validation is organised

Two suites.

• Standard: six sections (baseline, walkforward, montecarlo, costs, sensitivity, dsr) plus the sealed holdout.
• Institutional: the same six, plus every method in a fixed catalogue of 49 checks in seven layers. The engine's default is standard.

The seven layers, each with the question the code attaches to it:

Statuses. Each institutional check returns one of: pass, warn, fail, info (descriptive, no threshold), skip (not applicable) or error (could not be computed).

Conventions used by the institutional checks, quoted from THRESHOLDS.md:

Sharpe-type statistics use DAILY strategy returns (equity resampled to one value per calendar day with trading, flat days kept), annualised with sqrt(252) unless stated. "Trials" = the trial registry's count for this strategy lineage (falls back to the grid size). Every random draw is seeded from ValidationContext.seed.

In detail, the daily series is the last equity value on each UTC calendar day that has bars. The first day's return is measured against the starting capital. A day with no change is a zero return and is kept. If the account reaches zero, that day is −100% and later days are dropped.

sqrt(252) is used for the Sharpe even on data with more than 252 daily observations a year. The code notes that 24/5 FX and metals data has about 310 UTC days with bars per year; CAGR and track-record length use the data's own count, the Sharpe annualisation does not.

"Account-sized trades read at the starting capital". Several checks compare dollar profits between runs. On a compounding account a trade taken after the account tripled is three times the size of the first, so its dollars say more about when it happened than about the edge. Wherever the thresholds say a figure is read "at the starting capital", each default-sized trade's P&L is multiplied by capital / cash balance it met. Trades with an explicit qty, and overlapping trades, keep their dollars.

Time budget. Expensive checks scale themselves down to a soft budget (budget_s, default 120 seconds through the API, clamped to 10–900; the assistant's tool caps it at 45). For the whole institutional run:

deadline  = start + max(4 × budget, budget + 60 s)     checks not started by then are skipped
hard stop = deadline + max(budget, 30 s)               checks still running then are stopped

A check skipped for time makes the verdict "caution" at best.

Source. Suites and sections: backtest/validation.py:1-30, backtest/validation.py:51, backtest/validation.py:140-152. Catalogue, layers, statuses: backtest/checks.py:15-22, backtest/checks.py:33-112. Conventions: backtest/THRESHOLDS.md:7-11. Daily returns: backtest/stats.py:434-489, backtest/runner.py:1702-1718. Observations per year: backtest/stats.py:522-529. At-start scaling: backtest/research.py:142-226. Budget: backtest/institutional.py:259-288, engine/server.py:545, engine/server.py:1586, engine/server.py:4429.

4.2 The verdict

The verdict is computed from fixed rules and worded from fixed templates. Nothing in it is generated text. The API levels are trust, caution and do_not_trust; the app shows them as Holds up, Caution and Do not trust.

Standard suite. Level is do_not_trust if any veto fires, else caution if any caution fires, else trust.

Institutional suite.

• do_not_trust: any check with status fail in the layers integrity, execution, significance, overfitting, robustness or risk; or a standard veto that no check row already covers (no trades, a losing baseline).
• caution: any warn or error; a fail in the governance layer; a standard caution no row covers (too few trades, single-trial deflated Sharpe, reopened holdout); or any check skipped because the time budget ran out.
• trust: otherwise.

info and ordinary skip rows never move the verdict.

Source. Thresholds: backtest/research.py:424-442. Standard rules: backtest/research.py:477-488, backtest/research.py:518-663. Institutional fold: backtest/institutional.py:51-55, backtest/institutional.py:544-600. Display names: ui/static/guide.md:129. Guaranteed by tests: tests/test_validation.py:110-179.

4.3 The six core checks

Walk-forward

Question: do parameters chosen on one stretch of data still work on the stretch after it?

Procedure. With n bars, folds test windows and training share train:

oos_len   = floor(n × (1 − train) / folds)
train_len = floor(oos_len × train / (1 − train) + 0.5)
first test window starts at bar  n − folds × oos_len

Fold k's test window is the next oos_len bars; its training window is the train_len bars immediately before it (a rolling window, not anchored). With the defaults (4 folds, train 0.7) the four test windows are the last 30% of the bars, 7.5% each, and each training window is 17.5% of the bars.

For each fold, every parameter combination is run on the training window and the one with the best metric is kept (default netProfit; also profitFactor, winRate, avgTrade). That one combination is then run on the test window. Each run sees only its own window, starting from fresh capital, so indicators warm up inside the window.

Efficiency.

efficiency = (OOS profit / oos_len) / (train profit / train_len)

Both profits are the sum of trade P&L read at the starting capital. If the training window did not make money the fold's efficiency is undefined and is left out of the mean. meanEfficiency is the mean over the folds that have one. totalOosNetProfit is the sum of the test windows' dollar profits.

Threshold (THRESHOLDS.md): "mean efficiency >= 0.5 and out-of-sample profit > 0".

Verdict: below 0.5, not measurable, or OOS profit <= 0 is a veto.

Limits. It needs a parameter grid; without one it does not run. Test windows are short (7.5% of the data each by default), and a slow indicator's warm-up comes out of them. The grid is capped at 500 combinations.

There is also anchored walk-forward (institutional only): the same four test windows, but every training window starts at bar 0. Same threshold. Skipped when there are too few bars for 4 folds of at least 2 bars.

Monte Carlo trade resampling

Question: how much of the result was the particular order the trades arrived in?

What is resampled. The closed trades' P&L, in dollars, read at the starting capital. Not bars and not equity. Each path draws as many trades as the backtest had, with replacement, and replays them from the starting capital:

equity = capital
repeat (number of trades) times:
    equity += a randomly drawn trade P&L
    ruined  = true if equity <= 0 at any point
    track the path's peak and its largest drawdown in percent

Default 1,000 paths, seed 42, maximum 100,000 paths. The generator is a seeded 64-bit linear congruential generator, so the same trades and seed always give the same numbers.

What the outputs mean.

Threshold: "P(loss) <= 0.20 and risk of ruin <= 0.01".

Limits. Trades are drawn independently, so losing streaks that cluster in real markets are broken up; the real drawdown risk can be worse. It can only reshuffle trades that happened. It needs at least one closed trade.

Cost stress

Question: is the edge bigger than the costs?

Procedure. The strategy is rerun with every per-fill cost multiplied by the same number, by default 1x, 2x and 3x. Scaled together: commission_pct, slippage_bps, spread_bps (or the data spread through spread_data_scale), slippage_k and impact_c. Not scaled: financing and latency.

Threshold: "loses money at <= 2x costs -> fail; only above 2x -> warn; pass otherwise. skip when every scaled cost is zero".

Limit. If you set no costs, every multiple is the same run and the check proves nothing. The standard verdict says so as a caution.

Parameter sensitivity

Question: what does the ground around the chosen parameters look like?

Procedure. One full-sample run per combination in the grid. Each cell records the chosen metric (netProfit, profitFactor, winRate, avgTrade or sharpeRatio), its net profit, its trade count and its per-trade Sharpe. A combination that errors stays in the grid as a cell with no value.

Threshold: "under 50% of cells profitable -> warn". "Profitable" always means net profit above zero, whatever metric the grid was scored on.

Limit. Maximum 500 combinations. A single parameter spec may expand to at most 10,000 values.

Deflated Sharpe ratio

Question: after charging for every variant that was tried, is the Sharpe still better than the best of that many strategies with no edge?

Formula (Bailey and López de Prado, 2014), all Sharpes per observation, not annualised:

SR  = mean(returns) / std(returns)                               sample std
SR0 = sqrt(V) × ( (1 − γ) × Z(1 − 1/N) + γ × Z(1 − 1/(N·e)) )    expected best of N lucky trials
DSR = Φ( (SR − SR0) × sqrt(T − 1) / sqrt(1 − g3·SR + (g4 − 1)/4 · SR²) )

Threshold: DSR >= 0.95. Below is a veto.

What "trials" counts, and the two variants the platform shows.

1. Standard section ("this run's grid"). N is the number of combinations in the parameter grid passed to this validation. With no grid, N = 1 and the result is the probabilistic Sharpe ratio of the per-trade returns; the verdict adds a caution that only one trial was counted.
2. Institutional dsr row ("trial registry"). After the trial registry check has run, if the registry holds more variants than this run's grid, the deflation is redone with N = the registry's count and the row is replaced. V is unchanged, so only N moves. The registry counts every distinct parameter set ever validated for this strategy code on this symbol and timeframe, across sessions.

They differ because they answer different questions: "better than luck given today's search" versus "better than luck given every search ever run on this strategy and dataset". The second can only be equal or lower. The project's todo.md records an example of 82% versus 14.6% and notes that on screen it "reads as a contradiction"; it is an open design decision, not a bug.

Limits. N cannot see variants you tried by editing the file by hand before any validation, or variants tried under a changed copy of the code (the registry keys on the code hash). Neighbouring grid points are correlated, so the raw count over-charges somewhat. The formula assumes independent observations; overlapping trades break that.

Source. Walk-forward: backtest/runner.py:1269-1379 (window arithmetic 1298-1305, selection 1335-1347, efficiency 1355-1361). Anchored: backtest/overfit.py:1012-1095. Monte Carlo: backtest/research.py:69-139, called at backtest/validation.py:253-262. Cost stress: backtest/runner.py:1434-1499. Sensitivity: backtest/runner.py:1501-1566; caps backtest/runner.py:1390, backtest/research.py:284. Deflated Sharpe: backtest/research.py:309-409; trial count in the standard section backtest/validation.py:278-292; the registry recharge backtest/institutional.py:501-541. Thresholds: backtest/THRESHOLDS.md:41,37,23,50,31 and backtest/research.py:424-442. todo.md:25-28. Guaranteed by tests: tests/test_validation.py:45-86, tests/test_stat_ground_truth.py:775 (the paper's own numerical example), tests/test_stat_ground_truth.py:1223 (Monte Carlo against the exact binomial and gambler's-ruin answers).

4.4 Integrity: is the simulator right?

Notes on reading these:

• The truncation test allows a small end guard: within max(5 bars, 1% of the cut) of the comparison horizon, a trade that only the full run has ends the comparison instead of failing it, because causal scripts often stop opening trades a bar or two before the data ends. A trade that differs, or that exists only in the truncated run, fails anywhere.
• The delay test never fails, it only warns. The share kept is measured with trades read at the starting capital. If the baseline does not make money the row is info.
• The null-data sign test drops series on which the strategy did not trade. The p-value is P(X >= wins) for X ~ Binomial(m, 0.5) with m the number of series that traded. At least 5 series must run.
• The shadow engine models commission, slippage_bps, the ruin rule, spread, volatility slippage, latency, financing and size scale. Fill options it does not model (square-root impact) are rerun without and named in the row.
• Limits. These catch a class of leak, not all leaks. A strategy that leaks through a side dataset in data[...], or whose leak happens to survive a one-bar delay, can pass. Passing the known-answer test says the accounting is right for three simple cases.

Source. backtest/integrity.py:1-30 (module contract), constants backtest/integrity.py:50-67; check_known_answer 674; cases 624-636; check_lookahead_truncation 748 (margin 763); check_lookahead_delay 840-892; check_null_data 934-1016 (sign test 990-994); shadow_account 356; check_shadow_engine 1065; check_data_quality 1275 (spikes 1228, gaps 1245, grading 1290). backtest/THRESHOLDS.md:13-19. Guaranteed by tests: tests/test_integrity_checks.py:97-275 (each check is shown to fail a planted fault: a lying engine, a full-sample leak, a one-bar peek, a strategy that profits on noise, a miscounting engine, injected data faults).

4.5 Execution: would the fills have happened?

Round-trip cost in bps, as the break-even check counts it:

round trip = 2 × (commission_pct × 100 + slippage_bps + spread_bps)

Limits. The preset's one bar of latency is, in the code's words, "a stress rather than a forecast": on hourly or daily bars it is far slower than a real order. A cost you already set is kept and its row says it is already in the baseline. Capacity is an order of magnitude at best on tick-volume data. The asset class, and so the preset spread, is guessed from the symbol name.

Source. backtest/execution.py:1-22, component runs backtest/execution.py:212-224, backtest/execution.py:367-426; status rule backtest/execution.py:324-343; capacity backtest/execution.py:454-591. Break-even: backtest/stats.py:1212-1260, backtest/stats.py:1263. backtest/THRESHOLDS.md:21-25. Guaranteed by tests: tests/test_execution.py:260-347.

4.6 Significance: is it more than luck?

All of these use daily returns unless stated. n is the number of daily observations, SR the daily (not annualised) Sharpe, g3 the skewness and g4 the non-excess kurtosis.

Probabilistic Sharpe ratio (psr)

Question: how likely is it that the true Sharpe is above zero?

PSR = Φ( SR × sqrt(n − 1) / sqrt(1 − g3·SR + (g4 − 1)/4 · SR²) )

Threshold: "pass >= 0.95; warn >= 0.90; fail below." Limit: it treats the strategy as the only one ever tested.

Minimum track record length (min_trl)

Question: how much history does this Sharpe need before it is 95% likely to be above zero?

MinTRL = 1 + (1 − g3·SR + (g4 − 1)/4 · SR²) × (1.645 / SR)²      daily observations

Converted to years with this data's own observations per year. Threshold: "pass if the data covers it; warn otherwise." A Sharpe at or below zero needs infinite history and warns.

Autocorrelation-adjusted Sharpe (lo_sharpe)

Question: is the Sharpe flattered by returns that trend from day to day?

SR_annual = SR × η(252)
η(q) = q / sqrt( q + 2 × Σ (k = 1..10) (q − k) × ρ_k )

ρ_k is the lag-k autocorrelation of daily returns. With every ρ_k = 0 this is the usual sqrt(252). Threshold: "info; fail if the adjusted Sharpe is <= 0. When the autocorrelations make the correction undefined: warn (fail if the plain Sharpe is <= 0)."

Haircut Sharpe and multiple testing (haircut_sharpe)

Question: what is the Sharpe worth after a multiple-testing correction for the number of trials?

Harvey and Liu (2015), with the Benjamini–Hochberg–Yekutieli adjustment:

t      = SR × sqrt(n)
p      = 2 × (1 − Φ(t))                 two-sided, single test
c(M)   = 1 + 1/2 + … + 1/M
p_adj  = min(1, p × M × c(M))
t_adj  = Z(1 − p_adj / 2)
SR_adj = t_adj / sqrt(n)                then × sqrt(252) to annualise

M is the trial count: the registry's count, else the grid size, else 1. Only this strategy's own p-value is known, so it is treated as the most significant of the M, where the correction is tightest. The code calls this "the conservative end" of the method.

Threshold: "pass if the haircut Sharpe > 0 and the adjusted p < 0.05; fail otherwise. Also reports the t-statistic against the t > 3 hurdle." The t > 3 figure is reported, not graded.

White's Reality Check, Hansen's SPA and Romano–Wolf

These three read the same matrix: one column of daily returns per parameter variant. They share one set of 500 stationary-bootstrap resamples of the days (mean block length round(T^(1/3)) days, T = number of days), so they judge the same draws. The benchmark is cash: a return of zero.

Reality Check (reality_check). Question: could the best variant's mean return be this good by luck, given how many were searched?

V   = max over variants of  sqrt(T) × mean_k
V*  = max over variants of  sqrt(T) × (bootstrap mean_k − mean_k)      per resample
p   = share of resamples with V* >= V

SPA (spa). The same question, but each variant is divided by its own bootstrap standard deviation ω_k, and variants that are clearly worse than cash (mean below −ω_k / sqrt(T) × sqrt(2 ln ln T)) are left centred at their own negative mean, so they cannot inflate the luck distribution.

t_obs = max(0, max over variants of sqrt(T) × mean_k / ω_k)
p     = share of resamples whose max studentised statistic is >= t_obs

Romano–Wolf (romano_wolf). Question: which variants stay significant at 5% when the error rate is controlled across the whole grid? Stepwise: reject every variant whose bootstrap p-value against the maximum over the still-active variants is at most 0.05, remove them, repeat until nothing new is rejected. It also reports step-down adjusted p-values.

Thresholds: for reality_check and spa, "pass p < 0.05; warn p < 0.10; fail otherwise. skip under 20 trading days." For romano_wolf, "pass if the chosen variant does [stay significant]; fail otherwise. skip under 20 trading days." The "chosen variant" is the baseline's own parameters if they are a grid point, otherwise the grid's best by daily Sharpe.

Known limits, measured by the project's own simulations. These tests are supposed to reject a no-edge grid 5% of the time. Measured:

So SPA and Romano–Wolf pass a no-edge search somewhat more often than the 5% they claim, and more so on short or trending histories. The cause given in the tests is the noise in the variance estimate they divide by. Read a p just under 0.05 from these two as borderline. todo.md lists this under "Known limits (documented, not bugs)".

Two more limits: variants tried outside the grid are invisible to all three, and with a single variant there is no search to correct for.

Permutation test (permutation)

Question: did the real order of prices matter? The strategy is rerun on 100 price paths rebuilt from shuffled bar returns (gaps and intrabar shapes shuffled separately, so every bar is still a valid bar and the path ends at the same price).

p = (1 + number of shuffled paths with net profit >= the real one) / (1 + number of paths)

Threshold: "pass p < 0.05; warn p < 0.10; fail otherwise, and fail if the strategy lost money." At least 20 paths are needed for p < 0.05 to be reachable; with fewer because of time it is skipped. Limit: with 100 paths the smallest possible p is about 0.0099.

Bootstrap confidence intervals (bootstrap_ci)

Question: what range of Sharpe, annual return and drawdown is consistent with this backtest?

1,000 stationary block bootstrap resamples of the daily returns. The mean block length is chosen from the data by the Politis–White rule (near 1 day for independent returns, longer when they are dependent). On each resample: annual Sharpe (sqrt(252)), CAGR (using the data's own days per year) and maximum drawdown. The 95% interval is the 2.5th to 97.5th percentile.

Threshold: "pass if the Sharpe interval's lower bound > 0; warn if the median > 0; fail otherwise." Needs at least 10 daily returns.

Known limit, measured. On 600 simulated 500-day histories the 95% interval contained the true Sharpe 96.3% of the time on independent returns and 91.2% on autocorrelated returns (AR(1), φ = 0.3). So when returns trend, the interval is a little too narrow. todo.md records this as "covers about 91% (nominal 95%) on autocorrelated returns". It also cannot show an event the backtest never saw, and it ignores selection: a strategy picked from a grid has an interval centred too high.

Random-entry benchmark (random_entry)

Question: does the timing beat random timing with the same trades?

1,000 random strategies with the baseline's trade count, holding lengths, long/short mix and per-trade leverage, entered at uniformly random bars, filled at opens and charged the same per-fill costs. Each is scored as compounded growth Π (1 + leverage_k × r_k); the baseline is scored the same way. The result is the baseline's percentile (ties count half).

Threshold: "pass >= 95th; warn >= 90th; fail below, and fail if the strategy lost money." Square-root impact and financing are not charged to the random side.

Source. PSR: backtest/stats.py:74-95, backtest/stats.py:666-689. MinTRL: backtest/stats.py:98-112, backtest/stats.py:692-734. Lo: backtest/stats.py:115-140, backtest/stats.py:737-794. Haircut: backtest/stats.py:143-190, backtest/stats.py:797-833; trial count backtest/stats.py:558-569. Bootstrap machinery for the three grid tests: backtest/overfit.py:55, backtest/overfit.py:226-265; Reality Check backtest/overfit.py:268-274, backtest/overfit.py:752; SPA backtest/overfit.py:277-292, backtest/overfit.py:779; Romano–Wolf backtest/overfit.py:295-328, backtest/overfit.py:801; chosen variant backtest/overfit.py:197-221; 20-day minimum backtest/overfit.py:718. Permutation: backtest/stats.py:260-312, backtest/stats.py:889-974. Bootstrap intervals: backtest/stats.py:193-257, backtest/stats.py:982-1046. Random entry: backtest/stats.py:1049-1185. Thresholds: backtest/THRESHOLDS.md:27-38. Measured size and coverage: tests/test_stat_ground_truth.py:966-978, tests/test_stat_ground_truth_exhaustive.py:150-175, tests/test_stat_ground_truth_exhaustive.py:191-206, todo.md:37-45.

4.7 Overfitting: does it hold on data it never saw?

Probability of backtest overfitting (pbo, CSCV)

Question: how often does the variant that looked best in-sample turn out below the median out-of-sample?

Procedure. The matrix of daily returns (one column per variant) is cut into 16 equal contiguous blocks. For every one of the 12,870 ways to choose 8 blocks as in-sample and the other 8 as out-of-sample:

1. find the variant with the best in-sample Sharpe;
2. find its rank among all N variants out-of-sample; w = rank / (N + 1);
3. λ = ln(w / (1 − w)).

PBO = share of splits with λ < 0  +  half the share with λ = 0

A split landing exactly on the median counts as half, so pure noise reads 0.5 for any number of variants.

Threshold: "pass < 0.25; warn < 0.5; fail >= 0.5, except warn when the in-sample winner lost money out of sample in under 5% of the splits". Skipped with fewer than 2 distinct variants or fewer than 32 trading days.

Limit. PBO grades the choice, not the edge. If every variant carries the same real edge, the in-sample ranking is noise and PBO sits near 0.5 however good they all are. That is why a high PBO is downgraded to a warning when the winner almost never lost out of sample.

Combinatorial purged cross-validation (cpcv)

Question: if the variant is re-chosen on each training set, do the out-of-sample paths make money?

6 contiguous groups of days, 2 test groups per split: 15 splits, stitched into 5 full out-of-sample paths. Training days within the purge (the longest holding period, in days) on either side of a test group are dropped, plus an embargo of 1% of the days after it. The best variant by daily Sharpe on the training days is scored on the test groups.

Threshold: "pass if >= 75% of paths have Sharpe > 0 and the median > 0; warn if >= 50%; fail otherwise. skip without params." Limit: five paths is a small sample; 75% means 4 of 5.

Trial registry (trial_registry)

Question: how many variants have ever been tried on this strategy and dataset?

A file, validation_trials.json in the config folder, keyed by a hash of the strategy's code (pin line and whitespace ignored), the symbol and the timeframe. Every validation adds the distinct parameter sets it ran, plus the baseline's own parameters. The count is at least the size of the grid asked for, even if the grid failed to run. This count is the N used by the institutional deflated Sharpe, the haircut Sharpe and the minimum backtest length. Status is always info.

Limit. The key includes the code hash, so editing the strategy's logic starts a new count. Deleting the file resets everything.

Minimum backtest length (min_btl)

Question: is there enough data that the best of N lucky trials would not reach this Sharpe?

MinBTL (years) = ( E[max of N] / SR_annual )²
E[max of N]    = (1 − γ) × Z(1 − 1/N) + γ × Z(1 − 1/(N·e))

Threshold: "pass if the data covers it; fail otherwise. N = 1 needs no minimum (pass); a Sharpe <= 0 fails."

Sealed holdout (holdout)

Question: does it work on bars that played no part in building it?

Procedure. holdout_pct (0 to 0.5; the engine's default is 0, meaning no holdout) removes the last share of the bars, after any from/to window is applied, before anything else runs. Every other check sees only the remaining in-sample bars. open_holdout runs the plain strategy once on the held-back bars.

How openings are counted. Each opening is appended to validation_holdouts.json in the config folder with its time, the share, the bar count and the net profit. The key is a hash of the strategy's code (pin line and whitespace ignored), the symbol and the timeframe. The count is returned on every validation, opened or not.

Threshold: "opened more than once -> warn; opened and lost -> fail".

Limits. The counter is local and can be deleted. It keys on the code hash, so changing the code starts a new count. It keys on symbol and timeframe, not on the share, so shrinking the share exposes bars that used to be sealed without the counter moving. Nothing stops you running an ordinary backtest over the whole file; the holdout is sealed from the validation, not from you.

Out-of-time incubation (incubation)

Question: how has it done on bars added since it was first validated?

The registry stores the last bar seen at the lineage's first validation. On a later run with more data, the check measures only the equity after that point (or after the run on which these exact parameters were first tried, if later).

Threshold: "skip if none, if the parameters are new in this run, or if the registry holds no valid freeze point; pass if profitable with Sharpe > 0; warn if it passes on fewer than 20 trading days; fail otherwise."

Source. PBO: backtest/overfit.py:56-59, backtest/overfit.py:333-396, backtest/overfit.py:828-896. CPCV: backtest/overfit.py:60, backtest/overfit.py:401-457, backtest/overfit.py:919-972. Registry: backtest/overfit.py:52, backtest/overfit.py:481-486, backtest/overfit.py:550-612, backtest/overfit.py:615-707. MinBTL: backtest/overfit.py:462-476, backtest/overfit.py:975-1003. Holdout: backtest/validation.py:53-99, backtest/validation.py:189-237; row backtest/institutional.py:216-234. Incubation: backtest/overfit.py:1107-1179. Thresholds: backtest/THRESHOLDS.md:40-47. Guaranteed by tests: tests/test_stat_ground_truth.py:675 (PBO against explicit loops), tests/test_stat_ground_truth.py:1183 (PBO on correlated noise is one half), tests/test_validation.py:247-292 (holdout kept away from every check; openings counted).

4.8 Robustness: does it break easily?

Thresholds are quoted from THRESHOLDS.md, shortened where marked "…".

Limits. The regime tercile cut points are computed on the whole sample; THRESHOLDS.md calls them "a description, not a signal". cross_asset can only use datasets you have imported. The four stress windows are fixed dates; a dataset that does not cover them is not stressed by this check.

Source. Plateau: backtest/overfit.py:1192-1213, backtest/overfit.py:1216. Noise and jitter: constants backtest/overfit.py:61-63, grading backtest/overfit.py:1310-1328, checks backtest/overfit.py:1338, backtest/overfit.py:1436. Regimes backtest/overfit.py:1517; cross-asset backtest/overfit.py:64, backtest/overfit.py:1611; stress windows backtest/overfit.py:65-71, backtest/overfit.py:1666. Sub-periods backtest/stats.py:1509; outliers backtest/stats.py:1431. backtest/THRESHOLDS.md:49-58.

4.9 Risk and governance

Risk.

Daily VaR and CVaR, as implemented: VaR = −quantile(r, 1 − level), CVaR = −mean(r where r <= that quantile). With fewer than 100 days, the "worst 1% of days" is the single worst day, and the row says so.

Governance.

The bars' hash covers ts, open, high, low, close, volume in that order. The signature proves the report was not changed after export on this machine; it does not prove the backtest is right or that the data is what it says it is.

Source. Tail risk backtest/stats.py:315-345, backtest/stats.py:1647-1704; alpha backtest/stats.py:348-382, backtest/stats.py:1594-1644; turnover backtest/stats.py:1707-1757. Manifest backtest/report.py:100-112, backtest/report.py:124-172. Governance rows backtest/institutional.py:236-239. backtest/THRESHOLDS.md:60-69.