Separating Tweets from Croaks : Detecting Automated Twitter Accounts with Supervised Learning and Synthetically Constructed Training Data

Abstract: In this thesis, we have studied the problem of detecting automated Twitter accounts related to the Ukraine conflict using supervised learning. A striking problem with the collected data set is that it was initially lacking a ground truth. Traditionally, supervised learning approaches rely on manual annotation of training sets, but it incurs tedious work and becomes expensive for large and constantly changing collections. We present a novel approach to synthetically generate large amounts of labeled Twitter accounts for detection of automation using a rule-based classifier. It significantly reduces the effort and resources needed and speeds up the process of adapting classifiers to changes in the Twitter-domain. The classifiers were evaluated on a manually annotated test set of 1,000 Twitter accounts. The results show that rule-based classifier by itself achieves a precision of 94.6% and a recall of 52.9%. Furthermore, the results showed that classifiers based on supervised learning could learn from the synthetically generated labels. At best, the these machine learning based classifiers achieved a slightly lower precision of 94.1% compared to the rule-based classifier, but at a significantly better recall of 93.9%