Post-Pruning of Random Forests

Abstract: Abstract  Context. In machine learning, ensemble methods continue to receive increased attention. Since machine learning approaches that generate a single classifier or predictor have shown limited capabilities in some contexts, ensemble methods are used to yield better predictive performance. One of the most interesting and effective ensemble algorithms that have been introduced in recent years is Random Forests. A common approach to ensure that Random Forests can achieve a high predictive accuracy is to use a large number of trees. If the predictive accuracy is to be increased with a higher number of trees, this will result in a more complex model, which may be more difficult to interpret or analyse. In addition, the generation of an increased number of trees results in higher computational power and memory requirements.  Objectives. This thesis explores automatic simplification of Random Forest models via post-pruning as a means to reduce the size of the model and increase interpretability while retaining or increasing predictive accuracy. The aim of the thesis is twofold. First, it compares and empirically evaluates a set of state-of-the-art post-pruning techniques on the simplification task. Second, it investigates the trade-off between predictive accuracy and model interpretability.  Methods. The primary research method used to conduct this study and to address the research questions is experimentation. All post-pruning techniques are implemented in Python. The Random Forest models are trained, evaluated, and validated on five selected datasets with varying characteristics.  Results. There is no significant difference in predictive performance between the compared techniques and none of the studied post-pruning techniques outperforms the other on all included datasets. The experimental results also show that model interpretability is proportional to model accuracy, at least for the studied settings. That is, a positive change in model interpretability is accompanied by a negative change in model accuracy.  Conclusions. It is possible to reduce the size of a complex Random Forest model while retaining or improving the predictive accuracy. Moreover, the suitability of a particular post-pruning technique depends on the application area and the amount of training data available. Significantly simplified models may be less accurate than the original model but tend to be perceived as more comprehensible.