Definition
Overfitting happens when a model or trading rule is tuned so closely to a historical sample that it captures random noise instead of a repeatable pattern, so it performs well on past data and poorly on new data. The more parameters tried and the more combinations tested, the greater the risk.
How it works
Search itself creates false discoveries. If each of N independent tests of a rule with no real edge has a 5% chance of looking significant by luck, the chance that at least one does is 1 − 0.95^N, which is 99.4% for N = 100. The best of many variants is therefore a biased estimate of future performance.
Warning signs include results that depend on exact parameter values (a sharp peak, not a broad plateau), many adjustable parameters relative to the number of trades, no economic rationale, and collapse on untouched data. Fewer parameters, stability checks, walk-forward testing and a record of trials all help.
Formula / convention
P(at least one chance 'success') = 1 − (1 − α)^N, for N independent tests at significance level α
Illustrative example
Illustrative: a researcher tries 100 parameter combinations of a rule with no true edge and counts a result as significant at the 5% level. About 100 × 5% = 5 combinations are expected to pass by chance alone, so the winner's flattering in-sample numbers have no reason to repeat on untouched data.
What to watch for
Overfitting can be reduced but never eliminated, and a holdout period stops being independent once you keep adjusting the rule until it looks good there. Results selected from many trials should be discounted accordingly.