What you will learn
- Identify common reasons indicator signals fail.
- Apply a testing checklist covering costs, sample size and parameter stability.
Why signals fail
Indicators lag, so they signal late near turning points. In sideways markets trend-following rules produce whipsaws, a run of small losing trades, while mean-reversion rules can suffer in strong trends. No setting suits every market regime.
Random price series also produce shapes that look meaningful. A pattern that seems convincing in hindsight may occur just as often by chance, so a plausible story about a signal is not evidence.
A fair test
Write the rule completely before testing: entry, exit, parameters, instruments and period. Execute signals at the next available price rather than the bar that created them, include transaction costs, and compare with a simple benchmark such as holding the asset.
Keep part of the data unused during design and judge the final rule once on that unseen portion. Count the trades too, since a handful of trades says almost nothing about reliability.
Reading the evidence
Check neighboring parameters. If a 50-day average works but 40 and 60 days both fail, the result is probably noise. Robust ideas tend to degrade gradually as parameters change rather than collapse.
Trying many rules means some will look good by luck. Record how many variants were tested, expect live results to be weaker than the best backtest, and treat even a clean test as limited evidence, not proof.
Worked example: Costs can erase an edge
A moving-average rule earns 8.0% a year before costs on a fictional index but makes 24 round trips a year. If each round trip costs 0.30% in spread, commission and slippage, costs are 24 × 0.30 = 7.2%, leaving about 0.8%. If holding the index earned 5.0%, the rule trails it despite looking profitable before costs.
Check your understanding
A rule makes 12 trades and 9 win. If every trade were a fair coin flip, the chance of 9 or more wins is 299 ÷ 4,096. What is that, and what happens if 20 independent rules are tried?
Show answer and explanation
299 ÷ 4,096 ≈ 7.3%. With 20 independent rules, the chance that at least one reaches 9 or more wins by luck is 1 - (1 - 0.073) raised to the 20th power, about 78%. A good record from a small sample, picked from many tries, is weak evidence.