Deep learning to classify driver sleepiness from electrophysiological data

Abstract: Driver sleepiness is a cause for crashes and it is estimated that 3.9 to 33 % of all crashes might be related to sleepiness at the wheel. It is desirable to get an objective measurement of driver sleepiness for reduced sensitivity to subjective variations. Using deep learning for classification of driver sleepiness could be a step toward this objective. In this master thesis, deep learning was used for investigating classification of electrophysiological data, electroencephalogram (EEG) and electrooculogram (EOG), from drivers into levels of sleepiness. The EOG reflects eye position and EEG reflects brain activity. Initially, the intention was to include electrocardiogram (ECG), which reflects heart activity, in the research but this data were later excluded. Both raw time series data and data transformed into time-frequency domain representations were fed into the developed neural networks and for comparison manually extracted features were used in a shallow neural network architecture. Investigation of using EOG and EEG data separately as input was performed as well as a combination as input. The data were labeled using the Karolinska Sleepiness Scale, and the scale was divided into two labels "fatigue" and "alert" for binary classification or in five labels for comparison of classification and regression. The effect of example length was investigated using 150 seconds, 60 seconds and 30 seconds data. Different variations of the main network architecture were used depending on the data representation and the best result was given when using a combination of a convolutional neural network (CNN) and a long short-term memory (LSTM) network with time distributed 150 seconds EOG data as input. The accuracy was in this case 80.4 % and the majority of both alert and fatigue epochs were classified correctly with 85.7 % and 66.7 % respectively. Using the optimal threshold from the created receiver operating characteristics (ROC) curve resulted in a more balanced classifier with 76.3 % correctly classified alert examples and 79.2 % correctly classified fatigue examples. The results from the EEG data, both in terms of accuracy and distribution of correctly classified examples, were shown to be less promising compared to EOG data. Combining EOG and EEG signals was shown to slightly increase the proportion of correctly classified fatigue examples. However, more promising results were obtained when balancing the classifier for solely EOG signals. The overall result from this project shows that there are patterns in the data connected to sleepiness that the neural network can find which makes further work on applying deep learning to the area of driver sleepiness interesting.