Using Machine Learning to Predict Aggregate Excess Returns

Abstract: In this paper we examine whether standard linear regression and machine learning tools can be used to predict the time series of total returns in excess of the risk-free rate on the S&P500 and FTSE100 indices. We have virtually no success in predicting monthly returns. However, we do have some success in predicting annual returns. Fully 78 out of our 132 attempts at annual return prediction have a positive out-of-sample R2. Furthermore, we find that this translates to economic gains. We find that in 87 cases, long-only portfolios formed based on our models have a Sharpe ratio higher than a simple buy-and-hold portfolio. We achieve our best result when using US data from 1926 to 2019, to predict annual returns, using the random forest algorithm and a cumulative estimation window. We find that if we use this model to form a long-short portfolio, we have a Sharpe ratio of 0.996. This set up also has an out-of-sample R2 of 0.564. We use a Diebold-Mariano test to conclude that this performance, relative to benchmark predictions equal to the historical mean, is significant on the 1% level.