Recommendation of Text Properties for Short Texts with the Use of Machine Learning : A Comparative Study of State-of-the-Art Techniques Including BERT and GPT-2

Abstract: Text mining has gained considerable attention due to the extensive usage ofelectronic documents. The significant increase in electronic document usagehas created a necessity to process and analyze them effectively. Rule-basedsystems have traditionally been used to evaluate short pieces of text, but theyhave limitations, including the need for significant manual effort to create andmaintain rules and a high risk of complex bugs. As a result, text classificationhas emerged as a promising solution for extracting meaning from short texts,which are defined as texts limited by a specific character count or word count.This study investigates the feasibility and effectiveness of text classification inclassifying short pieces of text according to their appropriate text properties,based on users’ intentions in the text. The study focuses on comparing twotransformer models, GPT-2 and BERT, in their ability to classify short texts.While other studies have compared these models in intention classificationof text, this study is unique in its examination of their performance onshort pieces of text in this specific context. This study uses user-labelleddata to fine-tune the models, which are then tested on a test dataset fromthe same source. The comparative analysis of the models indicates thatBERT generally outperforms GPT-2 in classifying users’ intentions basedon the appropriate text properties, with an F1-score of 0.68 compared toGPT-2’s F1-score of 0.51. However, GPT-2 performed better on certainclosely related classes, suggesting that both models capture interesting featuresof these classes. Furthermore, the results demonstrated that some classeswere accurately classified despite being context-dependent and positionedwithin longer sentences, indicating that the models likely capture features ofthese classes and facilitate their classification. Both models show promisingpotential as classification models for short texts based on users’ intentions andtheir associated text properties. However, further research may be necessary toimprove their accuracy. Suggestions for enhancing their performance includeutilizing more recent versions of GPT, such as GPT-3 or GPT-4, optimizinghyperparameters, adjusting preprocessing methods, and adopting alternativeapproaches to handle data imbalance. Additionally, testing the models ondatasets from diverse domains with more intricate contexts could providegreater insight into their limitations.