I am currently working on demand forecasting of time series data for multiple products and have found that walk forward validation is the way to go, for implementing a train, validation and test split. I have some questions relating to this concept.
1) How big should my test set be? I currently split it so that it is the length of the forecasting horizon that I wish to have. In my case this is the last 26 data points in the complete data set, as this is the length of the forecast I wish to make.
2) How do I use a walk forward strategy to test a multi-step forecast? As I mentioned before, each time step I wish to make a forecast for the next 26 time steps. What I currently do then is take the complete dataset – the 26 points kept for the test set. I then split this subset into two further sets with a 50:50 split between train and validation, respectively.
I train the model on the train dataset and then I make a multi-step (26 step) forecast from the first date that appears in the validation dataset. I then measure the accuracy of these forecasts at both individual step level, and an average for the whole forecasting period. What I am specifically unclear on is, should I then add all of these points from the validation set to the train set and train the model again, so the next 26 points, or should I just add the immediate next step from the validation to the train set and repeat the process again?
Lastly I would pick the model with the highest accuracy at each step level and / or average accuracy over the whole multi-step period. Then I would test this final model against the final test set I took from the dataset earlier to get an average accuracy of the model and an accuracy measure per step, as the final evaluation of the trained model. Is this process correct, and can you see any flaws in it?