Short text classification with machine learning in the social sciences: The case of climate change on Twitter

Program (UNEP), the UN International Children’s Emergency Fund (UNICEF), the UN High Commissioner for Refugees (UNHCR), and the World Health Organization (WHO) [25, 26]. Not only the communication by these organizations on climate change, but also political debates, in general, are often complex and relate to multiple policy areas at the same time. The case of climate change debates on Twitter is not unique in this regard. This means that the findings reported in this article about the use of text classification methods for this case can also be applicable to other political debates. In this article, we make three main contributions to social science research using short texts. First, we identify relevant dictionary-based and ML methods, showing how these meth- ods might be employed for text classification. Second, we show, in the context of small labeled datasets exhibiting class imbalance, that supervised ML methods perform better than lexicons, in particular as class balance increases. Third, we find that traditional ML methods, such as logistic regression and random forest, perform similarly to sophisticated DL methods, whilst requiring much less training time and computational resources. In all, these contributions have important implications for the use of text classifiers in the social sciences, as we elaborate in the conclusions. The remainder of this article proceeds as follows. We begin by discussing the text classifica- tion problem that we address with automated text classification methods. We then review the

most commonly used automated text classification methods. Next, we present the dataset of UN agencies’ tweets and compare the performance of these methods in analysing the texts within this dataset. Finally, we summarize our findings and sketch avenues for future method- ological research on text classification. Automated text classification methods In this section, we review existing automated text-classification methods. These methods rely on a similar workflow consisting of the collection of data and various text preprocessing tech- niques, which we discuss in detail in the S1 Appendix. We start with the lexicon-based classi- fier. After this, we move on to the basics of the supervised-learning approach to classification. This is followed by an overview of relevant traditional ML algorithms. Finally, we discuss advanced neural classifiers. To describe the methods, we use the following notation. Boldface letters, such as a, denote vectors. Capital boldface letters, such as A, denote matrices and calligraphic letters, such as A, denote sets. Estimates are denoted as ^a. A vector transpose is denoted as aT. Lower indices, such as in aðiÞ j , are used to denote features, whilst upper indices denote observations. The prob- ability distribution of a variable a is denoted as p(a), and p(a|b) designates the conditional probability distribution of a given b has been observed, where we deliberately do not distin- guish between the random variables and their realizations. Lexicon-based classifier

Lexicon-based classification is the most widely used approach to text classification in the social sciences. It is an unsupervised technique that is based on the filtering or labeling of texts with the help of a lexicon, or a list of relevant key terms (e.g., [8, 10]). Typically, the lexicon that is used is created by an expert in the area of interest, who has knowledge of the terminology used in context in this area. When we use lexicon-based classification to categorize Twitter data from UN agencies, the content of tweets is tokenized (or split up into separate words) and then inspected. If one or more of the tokenized words in the tweet matches with key terms contained within the lexicon, then the tweet is classified as dealing with climate change. If no words match, then the tweet is PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 3 / 26