The Validity Layer
One deterministic check for each way a naive backtest number misleads, composed into a single honest verdict.
This page is the mechanics behind Built Not to Lie: one deterministic check per way a naive number misleads, composed into a single honest verdict. Every conclusion Warren draws, "this backtest is profitable," "this regime is bullish," "this Sharpe is real," passes through the validity layer before it reaches you. Each check is a pure function of its inputs that returns one of three verdicts, ok (inputs complete, premises hold), caution (usable but degraded, with reasons), or abstain (structurally untrustworthy, so the verdict is withheld and the reason given), plus plain-language reasons. Abstaining is a correct outcome, not an error.
Every check follows the four invariants of Built Not to Lie:
- Deterministic and label-free. Verdicts are pure functions of inputs, with no fitted model, so they are fully testable and immune to contaminated training data.
- Additive. A check annotates a result; it never rewrites the underlying numbers.
- Fail toward abstain. A check that could not run degrades the verdict; it never gets the benefit of the doubt. A money-touching "fail-open" is treated as the cardinal bug.
- Grounded, not self-reported. Anything checkable against data is checked in code, never by asking the model to critique itself, which rationalizes rather than verifies.
The checks
Each row is one deterministic check aimed at one specific way a number lies.
| Check | The way a naive number lies | What Warren does |
|---|---|---|
| Leakage / closed-candle | Reading a still-forming bar leaks the future and is not reproducible | All accuracy-critical reads drop the forming bar, erring toward dropping, and stamp candle-close time and data age |
| Coverage and abstention | A confident call built on one timeframe or stale data looks identical to a well-grounded one | A label-free verdict over input completeness, freshness, and internal consistency; thin or stale inputs abstain, with the specific reasons |
| Premise validity | A long-only strategy "tested" over what was actually a bear market was never exercised in its own regime | Infers the strategy's directional thesis from its trades, computes the window's realized regime, and abstains on a decisive mismatch: a wrong-regime pass cannot read clean |
| Beta vs. alpha | A long strategy "works" only because it rode the trend: +15% in a window that held +60% | Benchmarks against buy-and-hold, flags negative alpha, and will not certify leveraged beta as edge |
| Overfit statistics | The best of many trials looks impressive even on pure noise; raw Sharpe assumes normal returns | Probabilistic and Deflated Sharpe, enforced as a verdict rather than a prose caveat |
| Out-of-sample discipline | A strategy tuned and scored on the same window always looks good | A single run is labeled in-sample-only; "go live" is never offered without an out-of-sample leg |
| Cost realism | Modeling fees but not slippage or funding can show a profit that is a net loss live | States exactly which costs were modeled; flags thin-edge, high-churn strategies as cost-fragile |
Overfit-aware statistics
Warren implements the published Bailey and López de Prado statistics. The Probabilistic Sharpe Ratio is the probability that the true Sharpe exceeds a benchmark, correcting for sample length, skewness, and kurtosis. The Deflated Sharpe Ratio raises that benchmark to account for the number of trials searched and their variance: the multiple-testing correction that makes a cherry-picked best-of-fifty face the bar it deserves. Both are computed on the per-period (not annualized) Sharpe; feeding the headline annualized figure would saturate the statistic on pure noise, a common silent bug.
The "deflated" honesty flag. A result is labeled deflated only when a real search actually supplied a trial count and per-trial variance. Absent that, the statistic mathematically collapses to a single-trial probability, so Warren labels it "Probabilistic Sharpe (single-trial)" instead, and instructs every consumer to present it that way. The system refuses to claim more statistical rigor than it actually applied.
Premise validity and alpha
The premise check infers the strategy's directional thesis from its trades and compares it against the test window's realized direction, computed from closed candles over the backtest's actual date range: a wrong-regime pass cannot read clean. The companion beta-versus-alpha check asks a different question: did the return come from skill, or from simply riding the market?
Cost realism
A backtest that models exchange fees but not slippage or funding can show a profit that is a net loss once it trades live. The check first states exactly which costs were modeled (exchange fees are; slippage and funding are not), then reasons about fragility from aggregates rather than trusting the headline profit.
How the verdicts compose
Checks do not fire as independent alerts. A ledger rolls up the worst verdict; an abstain forces the result's deploy-readiness flags off and inserts its explanation at the top of the caveats. A premise-relevant concern, a real regime or statistical problem, blocks the go-live suggestion, while a mere "we could not verify" caution does not: it still degrades the verdict and travels with the caveats, but the go-live suggestion is left standing. Money movement itself remains confirm-gated regardless (see Execution). By the time a result reaches you it carries an explicit verdict, plain-language caveats, an honest name for its Sharpe figure, freshness stamps on its inputs, and a deploy option offered only once earned.
An earlier version of the product surfaced a single confidence number, and first-party analysis found it non-discriminative: over a large sample of decided paper signals it barely moved off one value and had essentially zero correlation with realized outcomes. The lesson is encoded everywhere: decisions come from explicit hard rules producing three-way verdicts, never from a threshold on one opaque score.
Black box. The numeric cutoffs, sample-adequacy floors, staleness multipliers, profit-factor tiers, mismatch thresholds, and the confidence bars on the Sharpe statistics are internal. What is documented is the contract: which check fires, what it means, and how it composes into the final verdict.