Can Hatescan Detect Antisemitic Hate Speech

Abstract: This thesis focuses on how well Hatescan, a hate speech detector built on the same Natural Language Processing and AI algorithms used in most online hate speech detectors, can detect different categories of antisemitism as well as whether or not it is worse at detecting implicit antisemitism than explicit antisemitism. The ability of hate speech detectors to detect antisemitic hate speech is a pressing issue. Jews have not only persevered through unparalleled historical oppression, but additionally, antisemitism is very much alive and kicking online, which poses not only a direct threat to individual Jews themselves (since there is a clear link between antisemitic expressions and antisemitic violence) but to the idea of liberal democracy itself. This thesis evaluated the efficacy of the hate speech detector, Hatescan, regarding its ability to detect antisemitism and to assess whether or not it was better or worse at detecting explicit antisemitism or implicit antisemitism, expressed in Swedish. Thus, the research questions posed for this thesis were: 1. How well does Hatescan detect antisemitism? 2. Is Hatescan equally efficient at detecting different categories of antisemitism? 3. Is Hatescan equally efficient at detecting implicit antisemitism and explicit antisemitism? To answer these questions, this thesis used the research strategy experiment, the data collection method documents, qualitative analysis methods (discourse analysis) for annotation, and quantitative analysis methods (descriptive statistics) for calculating performance metrics (precision, recall, F1-score, and accuracy). A dataset was created using three other previously existing datasets containing hate speech expressed in Swedish on Reddit, Flashback, and Twitter. The data collected was collected used search terms presumed to appear in antisemitic content. The datasets were created by the supervisor of this thesis and her research team for use in previous studies. These datasets were combined and made into one dataset (in a spreadsheet). Duplicates were deleted, adn each remaining sentence was annotated according to hatefulness, category of antisemtism and explicit versus implicit antisemitism. Each sentence was manually run through Hatescan’s web interface to generate a Hatescan output and said output was documented in the spreadsheet containing the data. Based on a threshold of 70% for generated Hatescan output, the Hatescan output for each sentence was annotated as either being a true positive, false positive, false negative, or true negative using IFS formulas in the spreadsheet. Precision, recall, and F1-score were calculated for the dataset as a whole, and accuracy rates were calculated for all categories of antisemitism as well as for explicit and implicit antisemitism. Results showed that while performance metrics on the antisemitic dataset (precision 0.93, recall 0.85, F1-score 0.89) were similar to the performance metrics in the development of Hatescan (precision 0.89, recall 0.94, F1-score 0.91), there were significant differences in accuracy between the different annotated categories in the dataset (accuracy ranging from 27 percent to 92 percent).