Keystroke Classification of Motion Sensor Data - An LSTM Approach

Abstract: With mobile phones often being used to type sensitive information, it is important that they remain secure and leak no information. One conceivable channel for information leakage, though, is the motion sensors accelerometer and gyroscope, sensors that require no permission to be used by an app. Do the data they produce contain information about the keys being typed? To answer this question, this thesis investigates whether, using LSTM networks, keystrokes can be classified as (1) either backspace or not backspace, and (2) any key on the keyboard, using only motion sensor data collected around the keystroke. Furthermore, the problems are investigated in three different cases, one where the models are built on a user-basis, one where they are built on a mobile phone brand-basis, and the last where they are built on a general basis, using data pertaining to all users and brands. The thesis finds that the motion sensor data do indeed contain information about the keys being typed. The different cases yield similar results for the backspace problem (1), while models built on a user-basis performs best in the more general problem (2). Training on a user-basis yields an EER of 0.11 and an F1-score of 0.61 for the backspace problem, and an Accuracy of 51 % and a macro-averaged F1-score of 0.32 for the more general problem, much better than naïve model performance.