A Transformer-Based Scoring Approach for Startup Success Prediction : Utilizing Deep Learning Architectures and Multivariate Time Series Classification to Predict Successful Companies

Abstract: The Transformer, an attention-based deep learning architecture, has shown promising capabilities in both Natural Language Processing and Computer Vision. Recently, it has also been applied to time series classification, which has traditionally used statistical methods or the Gated Recurrent Unit (GRU). The aim of this project was to apply multivariate time series classification to evaluate Transformer-based models, in comparison with the traditional GRUs. The evaluation was done within the problem of startup success prediction at a venture and private equity firm called EQT. Four different Machine Learning (ML) models – the Univariate GRU, Multivariate GRU, Transformer Encoder, and an already existing implementation, the Time Series Transformer (TST) – were benchmarked using two public datasets and the EQT dataset which utilized an investor-centric data split. The results suggest that the TST is the best-performing model on EQT’s dataset within the scope of this project, with a 47% increase in performance – measured by the Area Under the Curve (AUC) metric – compared to the Univariate GRU, and a 12% increase compared to the Multivariate GRU. It was also the best, and third-best, performing model on the two public datasets. Additionally, the model also demonstrated the highest training stability out of all four models, and 15 times shorter training times than the Univariate GRU. The TST also presented several potential qualitative advantages such as utilizing its embeddings for downstream tasks, an unsupervised learning technique, higher explainability, and improved multi-modal compatibility. The project results, therefore, suggest that the TST is a viable alternative to the GRU architecture for multivariate time series classification within the investment domain. With its performance, stability, and added benefits, the TST is certainly worth considering for time series modeling tasks.