Backtesting Traps: Overfitting, Survivorship, and Look-Ahead Bias

Three biases corrupt most backtests. Each has a specific signature and a specific prevention. Learn to spot them before they cost real money.

Overfitting

Overfitting occurs when a system fits noise in the sample rather than the underlying edge. Symptoms:

• Backtest Sharpe above 2.5 with many tuned parameters.
• Performance collapses on the smallest parameter change.
• The system was optimized repeatedly until it "worked."

Detection: perturb every parameter by 10%. If Sharpe drops by more than 25%, the system is overfit. Also test on a completely different market or symbol; a real edge generalizes, a fit system does not.

Prevention: limit free parameters to 3-4. Prefer robust parameters (e.g., a 50-period MA works, while a 47-period is suspicious). Always reserve 20-30% out-of-sample data and never touch it during development.

Survivorship Bias

Survivorship bias arises when the test universe excludes delisted or bankrupt symbols. A backtest of "buy the S&P 500 constituents in 2010" using today's membership overstates returns because it omits the firms that failed and were removed.

Detection: check whether your symbol list today matches the list at the backtest start. If you cannot reconstruct point-in-time membership, you have survivorship bias.

Prevention: use point-in-time constituent data. For US equities, this requires a paid data source (CRSP, Compustat, or a survivorship-free vendor). For futures, use continuous contracts constructed with proper roll logic, not spliced prices. Never backtest a stock-selection strategy on current index membership.

Look-Ahead Bias

Look-ahead bias uses information not available at the time of the trade. It is the most common and most damaging bias because it produces backtests that look perfect and fail live.

Common sources:

• Using the close of the current bar to decide whether to enter at the close.
• Using an indicator computed with the full series (e.g., a centered moving average) rather than the causal version.
• Using revised economic data; initial releases differ from revisions.
• Using today's split-adjusted price to compute a signal in the past before the split was known.

Detection: run the backtest bar-by-bar in strict temporal order, recomputing every indicator using only data up to and including the prior bar's close. Compare results to your fast backtest. Material divergence means look-ahead bias is present.

Prevention: code signals on a "shifted" basis, where any value used at bar t must be computed from data through bar t-1. Audit any use of "today's close" in an entry rule. Use as-reported economic data, never revised.

A Final Test

Run the system forward on live data without optimization for 50 trades. If live results fall far short of backtest expectations, one of the three biases is almost certainly present. Re-audit the code before adding capital. Backtests that look too good are usually wrong; the rare honest backtest looks modest.