QQ 301, 1 min read
All 49 checks at a glance
When the terminal validates a strategy it runs 49 checks and computes a verdict from them: trust, caution, or do not trust. The checks are grouped by seven questions.
This page lists every check and the bar it has to clear. The list is generated from the raw report of the run shown on this site, so it matches what the software produced. For how each check works and the maths behind it, read Validation, check by check.
A few checks record information and have no bar to clear. They are listed too.
Is the simulator right? Integrity, 6 checks
- Known-answer tests
- every case within 0.01% of capital; no trades returns capital exactly
- Look-ahead: truncation test
- trades entered before (cut - longest hold) identical at 60/75/90% cuts; within max(5 bars, 1% of the cut) of that, a trade only the full run has is an end guard, not a leak
- Look-ahead: one-bar delay test
- keeps >= 25% of net profit with every fill one bar later
- Null-data test
- no significant profit on 20 noise series: fail if profitable significantly more often than a coin (sign test p < 0.05, series with no trades are ties and dropped); warn if the median of the series that traded is positive
- Independent re-computation (shadow engine)
- equity curve and net profit within 0.01% of capital
- Data quality
- clean passes; gaps and spikes warn (holidays and news cause them); structural corruption on more than 0.1% of bars fails
Would the fills have happened? Execution, 7 checks
- Bid/ask spread fills
- realistic run profitable and keeps >= 50% of net profit
- Slippage and market impact
- realistic run profitable and keeps >= 50% of net profit
- Execution latency
- realistic run profitable and keeps >= 50% of net profit
- Financing and carry costs
- realistic run profitable and keeps >= 50% of net profit
- Cost stress
- profitable at 2x costs
- Break-even cost
- break-even >= 3x assumed cost (warn >= 1.5x)
- Capacity
- info: notional where impact halves profit
Is it more than luck? Significance, 12 checks
- Probabilistic Sharpe ratio
- PSR >= 0.95 (warn >= 0.90)
- Minimum track record length
- data covers MinTRL
- Autocorrelation-adjusted Sharpe (Lo)
- adjusted Sharpe > 0
- Deflated Sharpe ratio
- >= 95%
- Haircut Sharpe and multiple testing
- haircut Sharpe > 0 and adjusted p < 0.05
- White's Reality Check
- p below 0.05 (warn below 0.10)
- Hansen's SPA test
- p below 0.05 (warn below 0.10)
- Romano-Wolf stepwise test
- chosen variant significant at 5% after stepwise correction
- Permutation test
- p < 0.05 (warn < 0.10)
- Bootstrap confidence intervals
- Sharpe 95% lower bound > 0 (warn if median > 0)
- Monte Carlo trade resampling
- P(loss) <= 20% and ruin <= 1%
- Random-entry benchmark
- percentile >= 95th (warn >= 90th)
Does it hold on data it never saw? Overfitting, 8 checks
- Walk-forward analysis
- efficiency >= 0.5 and OOS profit > 0
- Anchored walk-forward
- mean efficiency >= 0.5 and OOS profit > 0
- Probability of backtest overfitting (CSCV)
- PBO below 0.25 (warn below 0.5); fail only if the winner loses out of sample in 5%+ of splits
- Combinatorial purged cross-validation
- 75% of paths Sharpe > 0 and median > 0
- Trial registry
- -
- Minimum backtest length
- data span at least MinBTL
- Sealed holdout
- opened once, and profitable
- Out-of-time incubation
- profitable with Sharpe > 0 on bars after the first validation
Does it break easily? Robustness, 9 checks
- Parameter sensitivity
- at least half the grid profitable
- Plateau score
- at least 70% of neighbours keep half the Sharpe (warn 40%)
- Noise injection
- >= 80% profitable and median keeps >= 50% (warn >= 60% profitable)
- Timing jitter
- >= 80% profitable and median keeps >= 50% (warn >= 60% profitable)
- Sub-period consistency
- >= 60% of years profitable, no year > 50% of total (fail < 40%)
- Regime breakdown
- profitable in >= 2 of 3 volatility regimes (warn 1)
- Outlier dependence
- profitable without the best 5 trades (warn: only without the best 1)
- Cross-asset generalisation
- at least half the other markets profitable (warn at least one)
- Stress-scenario replay
- no window loses more than 25% of the capital it started with
What is the return, really? Risk, 3 checks
- Tail and drawdown risk
- -
- Alpha versus buy-and-hold
- alpha t-stat > 2 (warn > 1)
- Turnover and exposure
- -
Can anyone check it? Governance, 4 checks
- Reproducibility manifest
- -
- Signed validation report
- -
- Independent review
- -
- Live versus backtest tracking
- -