The Effect of Optimization of Error Metrics

Abstract: It is important for a retail company to forecast its sale in correct and accurate way to be ableto plan and evaluate sales and commercial strategies. Various forecasting techniques areavailable for this purpose. Two popular modelling techniques are Predictive Modelling andEconometric Modelling. The models created by these techniques are used to minimize thedifference between the real and the predicted values. There are several different errormetrics that can be used to measure and describe the difference. Each metric focuses ondifferent properties in the forecasts and it is hence important which metrics that is used whena model is created. Most traditional techniques use the sum of squared error which havegood mathematical properties but is not always optimal for forecasting purposes. This thesisfocuses on optimization of three widely used error metrics MAPE, WMAPE and RMSE.Especially the metrics protection against overfitting, which occurs when a predictive modelcatches noise and irregularities in the data, that is not part of the sought relationship, isevaluated in this thesis.Genetic Programming, a general optimization technique based on Darwin’s theories ofevolution. In this study genetic programming is used to optimize predictive models based oneach metrics. The sales data of five products of ICA (a Swedish retail company) has beenused to observe the effects of the optimized error metrics when creating predictive models.This study shows that all three metrics are quite poorly protected against overfitting even ifWMAPE and MAPE are slightly better protected than MAPE. However WMAPE is the mostpromising metric to use for optimization of predictive models. When evaluated against allthree metrics, models optimized based on WMAPE have the best overall result. The results oftraining and test data shows that the results hold in spite of overfitted models.