Short text classification with machine learning in the social sciences: The case of climate change on Twitter

more time. In contrast, DL methods might be more useful when datasets are balanced and of a large scale, and when extensive computational resources are available. Conclusions In this article, we have compared the performance of various methods for the automated cate- gorization of short texts in a small and imbalanced dataset. This classification task frequently occurs in social science research when researchers analyse short texts such as sentences, para- graphs, tweets, or article abstracts. Moreover, it is often the case that the concepts that are being examined are only present in a small proportion of the overall dataset. We have reviewed the most widely used text classifiers and compared their performance in this challenging set- ting. For this purpose, we have used a novel dataset on the communication about climate change on Twitter by eight international organizations operating in the UN system in different policy areas. The article has made three main contributions to social science research engaging in auto- mated text classification. First, it has provided a systematic overview of central text classifica- tion methods, including lexicons, traditional machine learning, and deep learning. In doing so, the article helps to reduce the entry barrier for researchers seeking to use such methods in their work. Computer science is a vast field and it is important to filter for the state-of-the-art methods that can be used to advance various fields in the social sciences. Second, the performance assessment suggests that among the aforementioned classification

methods, lexicons are clearly outperformed by machine-learning methods. However, it is important to take into account the role of class imbalance. For instance, the use of a convolu- tional neural network was found to be the best choice when there is a balanced dataset, but for Fig 8. F1 score vs. balance ratio for the classifiers under consideration. Solid lines show the performance of traditional ML methods, dashed lines show that of DL-based approaches, while the dash-dotted line illustrates the performance of the unsupervised lexicon-based classifier. https://doi.org/10.1371/journal.pone.0290762.g008 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 21 / 26