Walk Forward Optimization (WFO)

What

WFO is a time series regression model validation strategy. There are two types in general what we follow is - * Select an initial training period and validation period of (3 months) * Train a model on the training period and validate on the validation period. * Do this for different periods. We currently run WFO on 4 periods - train till 2023 Q3 validate on 2023 Q4, train till 2023 Q4 validate on 2024 Q1, train till 2024 Q1 validate on 2024 Q2, train till 2024 Q2 validate on 2024 Q3.

Why

Since we have a historical decomp model we dont have any holdout set, so its very easy for us to overfit. This validation strategy gives us a good idea about the underlying model behaviour at a most granular level. While doing hyperparameter tuning or feature development this is very helpful.

Things to know

WFO is very compute exhaustive process as we need to train 4 different full models validate on them and then compute different types of metrics. Our model is at wibble level so, all our metrics are also calculated at that level. Granular metrics are huge so we dont track it on DataDog. On DataDog we track brand, state and country metrics. Key metrics to look at are MAPE, R2, Affine R2, Scaled R2 and relative R2. We have metrics for each period as well as average of all periods which helps us derive different insights about the model.