In-sample / Out-of-sample
The in-sample data is used during system development. The out-of-sample data is used during system validation. During development, the logic and parameters are repeatedly modified as a set of data is searched for patterns that precede profitable trading opportunities. By the time development is complete the results are good – they are always good – we do not stop playing with the system until they are good. System validation – the test of whether the system is detecting persistent patterns or has been fitted to the noise in the data – is evaluation of performance on data that has not been used during development.
Of course, the developer of the system can use whatever data he or she wishes for validation. But when the system is used for trading, the sequence is always develop, then trade. Actual trades are always out-of-sample. The best estimates of future performance come from tests made on out-of-sample data that immediately follows the in-sample data. Using data that precedes the IS data for OOS testing will over-estimate profit and under-estimate risk.
Recent Comments