If you take one habit away from backtesting, make it this: never judge a strategy on the same data you built it on. That single split, in-sample versus out-of-sample, is what separates a backtest that means something from an exercise in fooling yourself.
The two halves of your data
Split your history into two parts:
- In-sample data is what you build and tune the strategy on. You look at it, try ideas, adjust parameters, and iterate.
- Out-of-sample data is held back. The strategy never sees it during development. You only run on it once the strategy is finished, to find out how it does on data it had no chance to be fitted to.
The names are just “the data I worked on” and “the data I saved for the real test.”
Why in-sample results are optimistic by nature
Anything you tune to fit a fixed dataset will look good on that dataset. That is not evidence the strategy works; it is evidence you tuned it. The more parameters you adjust, and the more times you adjust them, the better the in-sample result looks and the less it means. This is overfitting: the parameters end up describing the noise in your sample rather than any durable pattern.
In-sample performance is the ceiling, not the expectation. It tells you how well you fit the past, which is not the question you actually care about.
Why out-of-sample is the real test
Out-of-sample data answers the real question: does the strategy generalize to data it was never shaped against? Because the strategy could not have been tuned to it, a good out-of-sample result is genuine evidence rather than an artifact of fitting. A strategy that shines in-sample and collapses out-of-sample was overfit, and you just found out cheaply instead of expensively.
How to split the data
A few common approaches, from simplest to most robust:
- Holdout. Reserve the most recent slice (say the last 20 to 30 percent) as out-of-sample. Build on the rest, test once on the holdout.
- Walk-forward. Repeatedly optimize on a window and test on the next, unseen window, then roll forward. This uses your data more efficiently and approximates re-optimizing as you go. See walk-forward optimization.
Common mistakes
- Peeking. Looking at out-of-sample results, then going back to tweak the strategy. The moment you do that, your out-of-sample data has become in-sample. You only get to use it as a clean test once.
- Reusing the holdout. Testing many strategy variants against the same out-of-sample set until one passes. That is overfitting to the holdout. The set is a one-shot check, not a tuning target.
- Too little out-of-sample data. A clean test on ten trades is noise. Make sure the out-of-sample period is large enough to mean something.
How Periscøpe helps
Periscøpe makes this discipline practical. Backtests are a deterministic, calendar-aware replay tied to an immutable strategy version, so you can run the same strategy on an in-sample window and then on a held-out out-of-sample window and trust that any difference is the strategy, not run-to-run noise. Optimization studies handle the in-sample parameter search as a set of backtest trials, and you then validate the chosen parameters on a later, unseen window using the same engine. The records for each run make it easy to compare in-sample and out-of-sample behavior honestly.