Common Backtesting Biases and Pitfalls

A backtest is a story told by historical data. Biases are the gaps between that story and what would have actually happened.

Why biases matter

Every retail backtest overstates returns. The question is by how much. Professionals assume a 30–50% haircut between backtested expectancy and live expectancy. Knowing the biases lets you anticipate the haircut instead of being shocked by it.

1. Survivorship bias

If your historical stock universe contains only companies that still exist today, you've filtered out all the bankruptcies and delistings. A "buy beaten-down stocks" strategy looks great when the losers that never recovered are missing from the data.

Example: Backtesting "buy stocks down 50% in a year" using current S&P 500 constituents will show miraculous recovery rates — because the ones that didn't recover are gone from the index.

Fix: Use point-in-time universe data that includes delisted and merged companies. For forex and futures, this bias is smaller (instruments rarely disappear), but contract rolls still matter.

2. Look-ahead bias

Using information that wasn't available at the time of the trade. Common forms:

• Using the close of the current bar to enter "at the close"
• Calculating an indicator that uses future data (e.g., centered moving averages)
• Using adjusted prices that include future splits
• Using a stock's full-year earnings that weren't released until Q4

Fix: Enter on the next bar's open. Audit every formula to confirm inputs use only data available at decision time.

3. Data-snooping bias (overfitting)

Testing many parameter combinations on the same data until something works. If you try 100 random strategies, ~5 will look profitable at the 5% significance level by pure chance.

Example: Optimizing EMA periods 5–50 in steps of 1, with stops 5–30, with targets 5–50 — that's thousands of combinations. The best is almost certainly noise.

Fix: Lock parameters before out-of-sample testing. Use walk-forward analysis. Apply the White's Reality Check or Hansen's SPA test for data-snooping corrections. Demand economic rationale before accepting a parameter set.

4. Selection bias

Picking only the trades or instruments that worked well in sample. Common forms:

• Backtesting only "high-volatility" days after seeing which were high
• Selecting only the futures contracts that performed best
• Filtering trades by an indicator that itself was tuned on the same data

Fix: Define all filters before testing. Don't inspect results and then add a filter.

5. Optimization bias

Tweaking every parameter until the equity curve looks perfect. The more parameters tuned, the more the system is fit to noise.

Rule of thumb: keep tunable parameters to 3 or fewer. Each additional parameter multiplies overfitting risk.

6. Survivorship in news filtering

If you filter out trades during "high-impact news," you're removing the exact periods that produce the largest moves. The filtered-out losses never appear, inflating expectancy.

Fix: Use an economic calendar fixed in advance, not one edited after the fact.

7. Transaction cost bias

Ignoring spread, commission, slippage, and swap. A 10-pip profit per trade on EURUSD becomes 7 pips net of a 1.5-pip spread and 1-pip slippage each way — a 30% reduction.

Fix: Always include realistic costs in the backtest. Be especially careful with high-frequency strategies, where costs dominate.

8. Look-ahead in regime filters

Using the entire-period VIX average as a regime filter is look-ahead. So is using the full-sample average true range to set stop distance.

Fix: Use rolling or expanding statistics. The system must work using only data available at each trade decision point.

9. Start-date bias

Choosing the start date to flatter the system. Backtesting a trend-following strategy starting in 2009 hides the painful 2011–2013 chop.

Fix: Test across multiple start dates. Report results from a fixed start date chosen before testing.

10. Curve-fitting stops and targets

Optimizing stop distance and target multiple to maximize profit factor will almost always produce a brittle system.

Fix: Use structure-based or volatility-based stops derived from market logic, not from optimization sweeps.

11. In-sample contamination

If you "just check" the out-of-sample period during development, it's no longer out-of-sample. Each peek degrades the test.

Fix: Lock the out-of-sample data file in a separate folder. Don't even look at it until the final rules are coded.

12. Liquidity illusion

Backtests assume you can fill any size at the printed price. In reality, large orders move the market.

Fix: Cap position size by average daily volume. Use volume-weighted slippage models.

13. Tick-data granularity bias

Backtests on 1-minute bars miss intra-bar wicks. A stop triggered inside the bar may have filled far from the printed low.

Fix: Use tick data for stops and limits. If unavailable, model slippage explicitly.

Practical audit checklist

• Universe includes delisted instruments
• No future data in any indicator
• Parameters were locked before out-of-sample test
• Out-of-sample data was untouched during development
• Realistic costs included
• Multiple start dates tested
• Walk-forward analysis performed
• No more than 3 tunable parameters
• Stops and targets have economic logic, not just best-fit numbers

Rule of thumb

If a backtest looks better than 1.5 profit factor with low drawdown and you're a beginner, suspect a bias. Real edges are modest. Real edges survive bias corrections. Phantom edges don't.

---

Next: walk-forward analysis — the gold standard for separating edge from overfitting.