Why backtests lie - and how we catch each lie
Every check in the audit exists because it caught a real failure in a real trading stack - ours. This page shows the receipts and the math.
The audit that started this
In early 2026 we ran a multi-strategy engine with a paper simulator that booked a 66% win rate. Deployed live, the same logic won 14% of the time. A forensic pass found the cause: the simulator granted limit fills at prices that had never traded. When we replayed the book against a realistic-fill control, 2.1% of the booked edge survived. A second forensic pass found that our labeling pipeline reported a median maximum adverse excursion of exactly 0.00% - trades that supposedly never went a tick against us - which is not skill, it is a time machine.
We rebuilt everything around one rule: suspect the harness, not celebrate the edge. The Reality Check is that rule, productized.
Check 1 - Fill verification
For US-listed equities and ETFs with timestamps, we sample fills and fetch the actual 1-minute SIP bar containing each one. If your booked price is outside that minute's [low, high] (with a 5bps tolerance for feed differences), the fill never happened. More than 10% impossible fills is fatal: the P&L was manufactured. When we ran our own historical fiction through this check, 71.7% of fills failed it.
Capability limit, stated plainly: v1 verifies US equities/ETFs only. Crypto, forex, and futures fills are reported as not-verifiable - never silently passed.
Check 2 - Artifact scan
- MAE ≈ 0: median adverse excursion of zero is the signature of after-the-fact entries.
- Sub-minute round trips at bar granularity mean the simulator invented the intrabar path.
- Exits before entries: log corruption or lookahead. Always fatal.
- 80%+ win rates over 30+ directional trades: almost always fill fiction or fat-left-tail bookkeeping.
- Impossible streaks relative to the log's own win rate.
Check 3 - Cost reality
We reprice every trade under conservative round-trip costs for your instrument tier. The liquid-ETF preset (1.5bps) is our own measured limit-at-NBBO execution; the other tiers use public spread and fee data. Most retail intraday edges are single-digit basis points - smaller than the spread they must cross. This is where they die, and it is better to learn that here than in a funded account.
Check 4 - Statistical honesty
Deflated Sharpe Ratio (Bailey & López de Prado, 2016): if you tried N variants and showed us the best one, your Sharpe must beat the expected maximum of N lucky draws before it means anything. You disclose N; we deflate. DSR below 0.5 caps the score - good numbers elsewhere cannot buy back unaddressed selection bias, because those numbers are exactly what selection bias inflates.
Bootstrap: 1,000 resamples of your trade sequence give confidence intervals on expectancy and a p-value against zero-edge.
IS/OOS time split: first half vs second half must both be positive. Regimes change; edges that lived only in one half of your history are memories, not strategies.
The verdict precedence
Findings are combined worst-first, and the worst evidence caps the score: fatal artifacts → negative gross expectancy → cost survival → major artifacts → IS/OOS → Deflated Sharpe → bootstrap → minor findings. A backtest with fabricated fills and a beautiful Sharpe is a fabricated backtest.
What the score cannot tell you
A 95 does not predict profits. It says: no detectable fiction, edge survives conservative costs, statistics survive skeptical corrections. The only test above that is a small live forward ledger, which is exactly what we tell every high scorer to go build. We built this tool because we needed it; we publish the limits because we got burned by tools that didn't.