Hybrid Variational Autoencoder for Clustering of Single-Cell RNA-seq Data : Introducing HybridVI, a Variational Autoencoder with two Latent Spaces

Abstract: Single-cell analysis means to analyze cells on an individual level. This individual analysis enhances the investigation of the heterogeneity among and the classification of individual cells. Single-cell analysis is a broad term and can include various measurements. This thesis utilizes single-cell RNA sequence data that measures RNA sequences representing genes for individual cells. This data is often high-dimensional, with tens of thousands of RNA sequences measured for each cell. Dimension reduction is therefore necessary when analyzing the data. One proposed dimension reduction method is the unsupervised machine learning method variational autoencoders. The scVI framework has previously implemented a variational autoencoder for analyzing single-cell RNA sequence data. The variational autoencoder of the scVI has one latent space with a Gaussian distribution. Several extensions have been made to the scVI framework since its creation. This thesis proposes an additional extension consisting of a variational autoencoder with two latent spaces, called hybridVI. One of these latent spaces has a Gaussian distribution and the other a von Mises-Fisher distribution. The data is separated between these two latent spaces, meaning that some of the genes go through one latent space and the rest go through the other. In this thesis the cell cycle genes go through the von Mises-Fisher latent space and the rest of the genes go through the Gaussian latent space. The motivation behind the von Mises-Fisher latent space is that cell cycle genes are believed to follow a circular distribution. Putting these genes through a von Mises-Fisher latent space instead of a Gaussian latent space could provide additional insights into the data. The main focus of this thesis was to analyze the impact this separation. The analysis consisted of comparing the performance of the hybridVI model, to the original scVI variational autoencoder. The comparison utilized three annotated datasets, one peripheral blood mononuclear cell dataset, one cortex cell dataset, and one B cell dataset collected by the Henriksson lab at Umeå University. The evaluation metrics used were the adjusted rand index, normalized mutual information and a Wilcoxon signed ranks test was used to determine if the results had statistical significance. The results indicate that the size of the dataset was essential for achieving robust and statistically significant results. For the two datasets that yielded statistically significant results, the scVI model performed better than the hybridVI model. However, more research analyzing biological aspects is necessary to declare the hybridVI model’s effect on the biological interpretation of the results.