In-sample and out-of-sample testing

Keep data the strategy has never seen, and use walk-forward testing to check that results hold up.

All courses · Lesson 3 of 6 · Advanced · 4 min

If you design and judge a strategy on the same data, the result flatters you. This lesson shows how to keep part of the history untouched, and how walk-forward testing repeats that check across time.

Two kinds of data

In-sample data is the history you use to build and tune the strategy: you look at its charts, choose indicators and adjust settings until the results look reasonable. Out-of-sample data is history you set aside and do not use for any of those decisions. It is only used afterwards, once, to see how the finished rules behave on prices they were never fitted to.

The out-of-sample period is normally the more recent one, so the test resembles how the strategy would be used going forward. If you peek at it and then change the rules, it stops being out-of-sample, because it has now influenced your choices.

A timeline split into an in-sample part and a later out-of-sample part, with rows of rolling walk-forward windows below, each an optimisation window followed by a test window.

Walk-forward testing

A single split gives one verdict, which can itself be lucky. Walk-forward testing repeats the process: optimise on one window, test on the window that follows, then slide everything forward and repeat. The test windows, joined together, form a series of results that were all produced on data the settings had not seen.

Worked example

Counting walk-forward windows

  1. You have 8 years of data, which is 96 months.
  2. Each round optimises on 24 months and then tests on the next 6 months.
  3. After each round the windows move forward by 6 months, so the first test window covers months 25 to 30, the next 31 to 36, and so on.
  4. The last test window covers months 91 to 96, so the test windows start at month 25 + 6k for k = 0 to 11, which is 12 windows.

The strategy is tested on 12 separate out-of-sample windows covering 72 months, rather than on one lucky split.

Choosing the split

There is no single correct split. A common rule of thumb is to keep somewhere between a quarter and a third of the history for out-of-sample testing, but the real constraint is trade count: the unseen period must contain enough trades for the error estimates from the sample-size lesson to be meaningful. A 20% hold-out with only 15 trades in it proves very little either way.

It also helps if the hold-out contains a mix of conditions, such as both trending and quiet periods. A strategy tested only on a calm stretch has not really been examined.

Strengths

  • A single split is simple and quick to run and explain.
  • Walk-forward testing gives several unseen-data results, so one lucky window matters less.
  • Walk-forward also shows whether the best settings stay stable as time passes.

Weaknesses

  • A single split gives one verdict, which can itself be luck.
  • Walk-forward needs a lot of data, and each test window has fewer trades.
  • Both lose their value if the results are used to adjust the rules and then reused.

How to read the outcome

Do not expect out-of-sample results to match in-sample ones; they are usually somewhat worse, because the in-sample numbers were partly fitted. What you want is the same sign and a similar order of magnitude. A large collapse, or results that swing wildly between windows, is a warning that the strategy depends on the exact period it was tuned on.

Key takeaways

  • In-sample data is used to build and tune the rules; out-of-sample data is kept untouched for an honest check.
  • Hold out the more recent period, and use it only once or the split stops being honest.
  • Walk-forward testing repeats optimise-then-test across rolling windows, giving several unseen-data results.
  • Expect somewhat worse results out of sample; look for the same sign and similar size, not a perfect match.

Sign in to save your progress and quiz results.