Short text classification with machine learning in the social sciences: The case of climate change on Twitter

RESEARCH ARTICLE Short text classification with machine learning in the social sciences: The case of climate change on Twitter Karina ShyrokykhID*, Max Girnyk, Lisa DellmuthID Department of Economic History and International Relations, Stockholm University, Stockholm, Sweden * karina.shyrokykh@ekohist.su.se Abstract To analyse large numbers of texts, social science researchers are increasingly confronting the challenge of text classification. When manual labeling is not possible and researchers have to find automatized ways to classify texts, computer science provides a useful toolbox of machine-learning methods whose performance remains understudied in the social sciences. In this article, we compare the performance of the most widely used text classifiers by applying them to a typical research scenario in social science research: a rela- tively small labeled dataset with infrequent occurrence of categories of interest, which is a part of a large unlabeled dataset. As an example case, we look at Twitter communication regarding climate change, a topic of increasing scholarly interest in interdisciplinary social science research. Using a novel dataset including 5,750 tweets from various international organizations regarding the highly ambiguous concept of climate change, we evaluate the performance of methods in automatically classifying tweets based on whether they are about climate change or not. In this context, we highlight two main findings. First, supervised machine-learning methods perform better than state-of-the-art lexicons, in particular as

class balance increases. Second, traditional machine-learning methods, such as logistic regression and random forest, perform similarly to sophisticated deep-learning methods, whilst requiring much less training time and computational resources. The results have important implications for the analysis of short texts in social science research. Introduction Contemporary society is characterized by the ever-greater production of textual data. Citizens post on social media, politicians give speeches which are commented upon live, and interna- tional organizations publish reports. Such activities provide social scientists with large amounts of freely available data, enabling them to advance theories in areas such as interna- tional relations, comparative politics, media studies, and public opinion. An example is the field of social media studies which is growing apace, as social networks have become an important aspect of people’s lives. The popularity of social media platforms means that they often serve as a means of communication for citizens living in distant places, civil society organizations, business representatives, and political institutions. Such societal and political actors tend to make use of social media platforms to exchange information with PLOS ONE PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 1 / 26 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Shyrokykh K, Girnyk M, Dellmuth L (2023) Short text classification with machine learning in the social sciences: The case of climate

change on Twitter. PLoS ONE 18(9): e0290762. https://doi.org/10.1371/journal.pone.0290762 Editor: Nebojsa Bacanin, Univerzitet Singidunum, SERBIA Received: December 30, 2022 Accepted: August 15, 2023 Published: September 29, 2023 Peer Review History: PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0290762 Copyright: © 2023 Shyrokykh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: The data underlying the results presented in the study are available here: https://zenodo.org/record/7633599#.Y- lbTS8w1qs.

the broader public. Social media has thus been defined as a social tool shaping new and unmanaged communication dynamics [1]. For example, the microblogging site Twitter is often used to package information in short messages [2–4]. In the study of short texts, researchers are often interested in categorizing or labelling texts according to their topic, relevance, or stance before carrying out further analysis. This text clas- sification process is a key challenge for two main reasons. First, manual classification of large numbers of texts might not be feasible and automated methods might be required. However, little is known about the relative performance of such methods in the context of social science research [5]. Second, concepts in the social sciences are often ambiguous and complex. This makes it difficult for automated classifiers to categorize texts based on those. A case in point is the topic of climate change and the risks it causes for societies and ecosys- tems. Over the past thirty years, framings of climate change as a policy challenge have prolifer- ated and differentiated across sectors and scales. Political debates about “climate adaptation”, “climate mitigation”, “carbon emissions”, “climate-induced displacement”, “disaster risk reduction”, and “climate vulnerability” all illustrate the complex nature of the concept [6, 7]. When classifying texts about climate change and its governance, researchers need to know how the concept of climate change is being referred to in the specific context in which it is being studied.

Several approaches to automated text classification are known. Lexicon-based methods are the most popular among social scientists [8–10]. These are based on lexicons or dictionaries of meaningful words related to a topic of interest, crafted by experts, and made available to the research community. Because such dictionaries are not always available, machine-learning (ML) algorithms are being increasingly applied. The two main categories of ML methods are supervised and unsupervised learning. In unsupervised ML methods, the computer may cluster the words in texts around unknown categories. These clusters, and the categories they might indicate, are then inter- preted by the researcher [11]. However, such approaches might not be a good choice when seeking to classify texts using labels that have been defined a priori. When the categories are known, supervised learning is more appropriate [12, 13]. Here, researchers first manually label a smaller dataset, which is then used to train a model that identifies the specified categories in a larger dataset in an automated way. Remarkably, in recent years, a subset of these methods, deep learning (DL), has achieved human-level performance carrying out certain tasks, such as image recognition [14] and text translation [15]. DL methods have proven very efficient at handling complex non-linear relationships within data, but require large amounts of advanced computations [16]. The choice of algorithm used for ML depends principally on the task at hand, but should

also be motivated by model performance. In this article, we compare several alternatives to find the best-performing algorithm. Drawing on the literature on text classification in the social sciences [5, 9, 17–19], we identify the most widely-used lexicon-based and supervised ML text-classification methods. We then investigate the performance of these methods in a typical social science research setting: a small labeled dataset with rare-event data. Our empiri- cal focus is on Twitter communication about climate change, which is studied in a rapidly growing body of interdisciplinary social science research [2, 20–24]. To delineate the empirical material, we make use of the Twitter communication of eight international organizations in different policy areas that are known to be central in communi- cating about climate change, and which are comparable in their communication as they are all part of the United Nations (UN): the Food and Agriculture Organization (FAO), the Office for the Coordination of Humanitarian Affairs (UNOCHA), the UN Development Programme (UNDP), the UN Office for Disaster Risk Reduction (UNDRR), the UN Environmental PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 2 / 26 Funding: This work was supported by the project: “Glocalizing Climate Governance: The role of Integrated Governance for a Just and Legitimate Adaptation to Climate Risks (GlocalClim)” funded by Formas under grant number 2018-01705 (Svenska Forskningsrådet Formas, Grant/Award

Number: 2018-01705). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist.

Program (UNEP), the UN International Children’s Emergency Fund (UNICEF), the UN High Commissioner for Refugees (UNHCR), and the World Health Organization (WHO) [25, 26]. Not only the communication by these organizations on climate change, but also political debates, in general, are often complex and relate to multiple policy areas at the same time. The case of climate change debates on Twitter is not unique in this regard. This means that the findings reported in this article about the use of text classification methods for this case can also be applicable to other political debates. In this article, we make three main contributions to social science research using short texts. First, we identify relevant dictionary-based and ML methods, showing how these meth- ods might be employed for text classification. Second, we show, in the context of small labeled datasets exhibiting class imbalance, that supervised ML methods perform better than lexicons, in particular as class balance increases. Third, we find that traditional ML methods, such as logistic regression and random forest, perform similarly to sophisticated DL methods, whilst requiring much less training time and computational resources. In all, these contributions have important implications for the use of text classifiers in the social sciences, as we elaborate in the conclusions. The remainder of this article proceeds as follows. We begin by discussing the text classifica- tion problem that we address with automated text classification methods. We then review the

most commonly used automated text classification methods. Next, we present the dataset of UN agencies’ tweets and compare the performance of these methods in analysing the texts within this dataset. Finally, we summarize our findings and sketch avenues for future method- ological research on text classification. Automated text classification methods In this section, we review existing automated text-classification methods. These methods rely on a similar workflow consisting of the collection of data and various text preprocessing tech- niques, which we discuss in detail in the S1 Appendix. We start with the lexicon-based classi- fier. After this, we move on to the basics of the supervised-learning approach to classification. This is followed by an overview of relevant traditional ML algorithms. Finally, we discuss advanced neural classifiers. To describe the methods, we use the following notation. Boldface letters, such as a, denote vectors. Capital boldface letters, such as A, denote matrices and calligraphic letters, such as A, denote sets. Estimates are denoted as ^a. A vector transpose is denoted as aT. Lower indices, such as in aðiÞ j , are used to denote features, whilst upper indices denote observations. The prob- ability distribution of a variable a is denoted as p(a), and p(a|b) designates the conditional probability distribution of a given b has been observed, where we deliberately do not distin- guish between the random variables and their realizations. Lexicon-based classifier

classified as not dealing with climate change. The number of tokenized words which need to match for positive classification is a design parameter. There are many possible ways to classify tweets with respect to the topic of climate change. However, there is no standard lexicon for this type of classification. For this article, in order to benchmark the ML methods with a reasonable classifier, we select the lexicon from [3]. It con- sists of the following key terms: “climate”, “climatechange”, “globalwarming”, “climaterealists” and “agw”(an abbreviation for “anthropogenic global warming”). The lexicon has a decent per- formance and constitutes a good baseline for our empirical performance assessment. Supervised approach to text classification An efficient alternative to lexicon-based classifiers is the use of supervised ML methods. This class of methods requires a manually labeled dataset D (see Eq (1) and Fig 3 in the S1 Appen- dix). By fitting a set of N examples given in this labeled dataset, the methods obtain a model that approximates the predictor function. To fit the model, we choose a mapping ^f ð�Þ that minimizes the empirical average loss. This means that, in fitting the model, the computer attempts to minimize the difference between the predicted value and the actual observation. This can be formally expressed as: Lð^f Þ ¼ 1 N X N i¼1 ‘ ^f ðxðiÞÞ; yðiÞ � � ; ð1Þ where ‘ð^f ðxðiÞÞ; yðiÞÞ is some chosen loss function that reflects the closeness of the model pre- diction ^f ðxðiÞÞ for features x(i) to the true label y(i) for observation i.

The model ^f ð�Þ, being a function of the features x, is also a function of a set of its own parameters w = [w0, . . ., wK−1]. The fitting of the model is therefore carried out by searching the parameter vector w that minimizes the average loss in Eq (1). This process is referred to as the training of the model. By monitoring loss Lð^f Þ we can track the in-sample performance of the model, or more specifically how closely ^f wð�Þ corresponds with the data to which it is being fitted. Because the model learns the “average” mapping from features into labels, we also expect the model to have a decent out-of-sample performance (i.e., performance on new and as-yet- unseen examples). Unfortunately, the in-sample performance of a model might not be indica- tive of its out-of-sample performance, as we shall see below. If the model is too simple (i.e., it has too few parameters w to capture the behavior of the data), it is unable to fit the data well and has poor in-sample performance. This is known as underfitting and is an indicator that the model is poor in general. To avoid underfitting, more complex models have to be considered. The true predictor f(�) can be a complicated relation between features and labels. To approximate it well, the model ^f wð�Þ may sometimes need to be complicated and include many parameters K. For example, a linear model of this kind could be given by the following expression: ^f wðxÞ ¼ w0 þ w1φ1ðxÞ þ � � � þ wK 1φK 1ðxÞ, where φk(x) is some function of features x for k = 1, . . ., K − 1. More complicated models (e.g.,

polynomials) might involve even larger numbers of parameters. The more complex the model ^f wð�Þ, the greater the possibility to decrease the empirical loss Lð^f Þ in Eq (1) and improve its in-sample performance on the given dataset D. However, when the number of parameters becomes too high (relative to the number of examples N in the data- set), the training process might start confusing the noise in the data with the behavior of the feature variables. The model ^f wð�Þ becomes tied to particular examples in the dataset and does not generalize to unseen data. That is, the in-sample performance of the model on the dataset PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 4 / 26

becomes highly misleading and does not reflect its out-of-sample performance, which is known as overfitting. Thus, a general rule for picking the model is the following: the model should be complex enough to overcome underfitting, but simple enough to not reach overfitting. To detect overfitting, the entire dataset D can be split into two parts: a training set Dtrain and a validation set Dvalid. This makes it possible to validate the out-of-sample performance of a model on the validation set. Then several models can be fitted to the training set, while also val- idating their out-of-sample performance using the validation set. This validation can be used as a basis for selecting a model that is complex enough to generalize to new examples and per- forms the best in terms of average loss among all other considered models. Following this, the best model can be retrained using the entire dataset (minus the test set), whilst also ensuring that it generalizes well when applied to as-yet-unseen data. This approach makes it possible to validate the models and select the best one in terms of the out-of-sample loss. However, it might still not be possible to assess the actual performance of the model on as yet unseen data. This is because the process of model selection contaminates the dataset, injecting a dependency between the model and the data based on which it was selected. Hence, the validation loss is not representative of the actual performance on yet unseen data. To tackle this problem, it is necessary to make yet another partition of the dataset

D. That is, prior to cutting subsets Dtrain and Dvalid from the dataset D, another subset of data, called a test set Dtest is withheld from the validation set. This test set is not touched during the training and validation, and is used purely for assessing the classifier’s performance on unseen data. Traditional machine-learning classifiers By traditional ML classifiers we mean a collection of ML algorithms which have previously been developed for classification tasks. Those are built on different principles and have differ- ent computational complexities. Their performance depends on the task at hand. In this sec- tion, we will review the most popular ML classifiers and specify the variants of the classifiers that will be used for our subsequent performance comparisons. In our analysis, we use the Sci- Kit-Learn implementations [27] of the traditional ML algorithms, whereas for the deep neural classifiers, we use the TensorFlow implementations set up via the Keras API [28]. For each of the methods, we test the three available vectorizers, as well as optimize the number of features to maximize the classifier’s performance. Logistic regression. Logistic regression (LR) is a classification algorithm that was invented in [29] and has been widely applied in text classification [30, 31]. The classifier mod- els a binary random variable representing the label y given the data x. The model, parameter- ized by weights w, is given by ^f wðxÞ ¼ s w0 þ X M j¼1 wjxj ! ; ð2Þ where σ(�) is the sigmoid function representing the conditional probability of a positive class

label given the observed features x1, . . ., xM. An example of a LR classifier in action is shown in Fig 1 on exemplary synthetic data. The two classes are separated by a decision boundary, which is determined by the equal-probability line on the sigmoid-based probability map (see the black solid line on Fig 1). For all data points with probability above the decision threshold we predict ^y ¼ 1, while for the rest we predict ^y ¼ 0. PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 5 / 26

The advantages of the LR classifier are its simplicity, efficient training, interpretability and the absence of assumptions about the class distribution. The downsides are overfitting for a large number of features (compared to the number of observations) and sensitivity to multicol- linearity among predictors. A classifier is characterized by a set of hyperparameters which refer to parameters related to the model’s architecture and whose values are set before the learning process begins. This is in contrast to the model’s own parameters which are learned during the training. The SciKit- Learn implementation of the LR classifier has several hyperparameters, including regulariza- tion strength and maximum number of iterations. For our numerical performance assessment, these are tuned to get the best classification performance. Support vector machines. Support vector machines (SVMs) were first proposed in [32] for linearly separable problems and later generalized to non-linear problems in [33]. SVMs have been frequently used for the classification of political texts [12, 13, 18]. They are some- times seen to be the classification method to be used for short texts in social sciences [8]. Unlike the LR method which provides a probability as its output, the output of an SVM gives the direct prediction of the label. SVM classification works by fitting a hyperplane that separates the classes and maximizes the separation margin [32]. Support vectors refer to those samples that are the closest to the

separating hyperplane. These are the only samples that impact the SVM training, as they are most likely to cause misclassification. The margin is given by the length of the projections of the supporting vectors onto the vector of parameters w which is perpendicular to the separat- ing hyperplane. The average loss is given by Eq (1) with a hinge loss function. Fig 1. Illustration of a logistic regression classifier. The sigmoid-based surfaces shows the probability of the positive class given the observed data. https://doi.org/10.1371/journal.pone.0290762.g001 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 6 / 26

The above intuition only works for cases where classes are linearly separable (see Fig 2a). In the general case, when classes may overlap in such a way that it is not possible to fit a separat- ing hyperplane between their data points, the SVM principle can be extended to soft margins. This allows some observations to reside on the other side of the hyperplane [34]. Furthermore, a non-linear transformation ϕ(�) can also be applied to the data, expanding the feature space to a higher dimension in which there may be a linearly separating hyperplane. This transforma- tion can be done, e.g., by adding higher-order polynomial terms. However, a problem with this approach is that it might explode the computational complexity of the algorithm and lead to overfitting. In order to overcome this problem, the “kernel trick” can be invoked. Namely, by means of Lagrangian duality, the maximization of the soft margin is reformulated in terms of scalar products between data points. There are kernel functions k(�) where k(x(i), x(j)) = ϕ (x(i))ϕ(x(j))—e.g., polynomial, Bessel, radial basis functions (RBFs). This means that it is possi- ble to skip computing the data transformation ϕ(x). Instead, only computing the kernel func- tion will give the dot products in the transformed feature space for the optimization. The obtained separating hyperplane translates back to the original feature space as a non-linear decision boundary (see Fig 2b that illustrates an SVM with an RBF kernel on synthetic data). The advantages of the SVM classifier are its operation efficiency with high-dimensional

data, ability to model non-linear decision boundaries, and efficient handling of small datasets. The downsides are its sensitivity to the choice of the kernel function, high computational com- plexity for very large datasets, and the lack of probabilistic interpretation (unlike in LR). Because it is known that text classification is usually linearly-separable [12], we use an SVM implementation with a linear kernel for our empirical analysis. The regularization hyperpara- meter, defining the “softness” of the margin, is tuned to maximize the performance of the SVM. Random forest. Random forest (RF) classifiers [35, 36] are widely used for text classifica- tion (see, e.g., [37, 38]). The algorithm uses the so-called decision tree approach [39, 40]. The latter relies on the intuition that, instead of seeking a complicated mapping function ^f ðxÞ, one can partition the feature space into disjoint regions and fit a simpler model in each of those. This is done during training by recursively splitting the space of possible values for each feature at a certain threshold. For example, at a given iteration, the entire input space is split into a pair of half-spaces {x : xj � tj} and {x : xj < tj}, where tj is a threshold for a certain feature j. The split is done in a greedy way to minimize some loss function (e.g., misclassification error, Gini impurity, or entropy), without looking ahead at future splits. This process is repeated on each of the half-spaces recurrently until a stopping criterion is fulfilled (e.g., there is no information

gain from further splits). In this way, a binary “decision tree” is constructed, which sets the thresholds for the classification of new samples. Once the training is done, a new observation (x, y) is passed through the entire decision tree. As it passes through the tree, the observation experiences a sequence of learned condi- tional statements (greater or lower than a threshold) for each feature xj where j = 1, . . ., M. At the bottom of the tree (in a leaf node, where there is no further split), the prediction is obtained as the majority class of the data points satisfying all the conditions along the path. Illustrations of predictions with decision trees are shown as one block in Fig 3. Each circle depicts a condi- tional statement over a feature with respect to a threshold. The thick lines depict the paths of observations as they travel through the decision tree, satisfying all the conditions on their way. The numbers below the trees indicate the predicted classes. An issue with decision trees is that they typically overfit training data. A widely-used way to improve generalizability is to average the predictions of a number of decision trees by means of bagging [41]. With bagging, several bootstrap training sets D1; D2; . . . ; DB are PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 7 / 26

Fig 2. Illustration of a support vector machine classifier. A hyperplane is fitted to separate the two classes and maximize the margin. Linear kernel (a) is used for the case of linearly separable classes, while radial basis function kernel (b) is used for the case with class overlap. https://doi.org/10.1371/journal.pone.0290762.g002 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 8 / 26

randomly sampled (with replacements). A decision-tree model is then fitted to each of these training sets. The predictions from the models are combined, e.g., using majority voting over decision trees in the forest. This improves the model’s generalizability by reducing the variance of the classifier, without increasing its bias. The RF approach is an extension of bag- ging that also randomly selects subsets of features used in each data sample. Fig 3 illustrates an RF classifier in action. A total of B decision trees are formed by considering various sub- sets of available features. Each decision tree makes its own prediction on its own subset of the training data. Then majority vote defines the final prediction label of the random forest classifier. The RF classifier has the advantages of good interpretability, possibility to assess the impor- tance of input features, and efficient avoidance of overfitting with many trees. The main disad- vantage of this method is its computational complexity and slow speed with a large number of trees, which limits its applicability for real-time operation. For our empirical analysis, we adjusted the maximum depth of the decision trees and the number of estimators in order to optimize the RF classifier’s performance. k nearest neighbors. The k nearest neighbors (KNN) algorithm [42] is a simple classifier which has been widely used by researchers for text categorization (see, e.g., [43, 44]). In con- trast to other ML algorithms, KNN does not have a training phase. Instead the training set is

simply stored and the actual computation is deferred to the prediction phase. In a nutshell, given an unlabeled observation x, the algorithm calculates the distances d(x, x(i)) to training observations x(i), for i = 1, . . ., N, in the feature space and finds k nearest neighbors to x. The classes of these neighbors are used to weigh the available classes for the given observation. That is, majority voting is performed on the observations within the set of k nearest neighbors. Fig 4 illustrates a KNN classifier in action on synthetic data. Fig 3. Illustration of a random forest classifier. The classifier consists of a collection of decision trees, each making its own prediction regarding the class of an unseen data point. The predictions are then used in a majority voting for producing the final prediction. https://doi.org/10.1371/journal.pone.0290762.g003 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 9 / 26

In our analysis, we use KNN with uniform weights. Because of the majority voting proce- dure, we set the hyperparameter k to an odd number. We sweep k and another hyperpara- meter, leaf size, to optimize the KNN performance. Naïve Bayes. Naïve Bayes (NB) is a probabilistic classifier whose basic idea is to use the joint probabilities of words and categories to estimate the conditional category probabilities given a data point [45]. It has been successfully used for text classification [46, 47]. It is called “naïve” because it makes an assumption that features are conditionally independent given the label. This assumption simplifies the computations carried out by the NB classifier, when com- pared to a non-naïve Bayes classifier, because the NB classifier does not use word combina- tions as predictors. Given an observation x, such as a short text, the NB classifier assigns the most probable cat- egory ^y according to ^y ¼ argmaxc2C fpðcjxÞg ¼ argmaxc2C fpðxjcÞpðcÞg; ð3Þ due to Bayes’ theorem. The term p(x) is not dependent on class c and hence skipped. Since it is difficult to estimate p(x|c), the naïve Bayes assumption is made. Namely, we factorize this con- ditional distribution as p(x|c) = p(x1|c)p(x2|c) � � � p(xM|c), which simplifies the computations. Moreover, p(c) is estimated as the fraction of the tweets in class y in the training set. Mean- while, p(xj|c) is estimated as the relative count of feature xj in class c to all features, with applied Laplace smoothing. The advantages of the NB classifier are its simplicity, good scalability, quick training and

insensitivity to irrelevant data. The disadvantages of the method are the assumption of inde- pendent predictor features and zero-probability problem for features present in the test set, but not occurring in the training set. Fig 4. Illustration of a k nearest neighbors classifier. For an unseen data point, the distances towards its k nearest neighbors are computed, and the majority class gets assigned. https://doi.org/10.1371/journal.pone.0290762.g004 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 10 / 26

There are several versions of the NB classifier (e.g., Gaussian, Bernoulli, multinomial). The operation of a Gaussian NB classifier on exemplary data is demonstrated in Fig 5. Studies have reported that a multinomial mixture model shows improved performance scores on several data collections, when compared to other commonly used versions of the NB approach [46]. Because of this, for our empirical analysis, we use a multinomial NB classifier with the smooth- ing hyperparameter tuned to achieve the best performance. Deep learning Deep learning (DL) is a prominent supervised ML framework for data-driven predictive modeling. Despite its promising classification capacities, DL continues to be a much less used approach to text classification in social science, when compared to its use in such fields as com- puter science. A DL classifier can be thought of as a black box: data go in, decisions come out (see Fig 6a). Inside the black box there are a large number of parameters that are learned dur- ing the training. In this way, the DL model approximates a predictor function that describes the observed data. Because of the way its operations resemble those of the human brain, the broader class of methods to which DL belongs is often referred to as artificial intelligence (AI). A central element of DL is the concept of an artificial neural network (NN) which describes the inner-workings of the DL black box. It consists of a layered structure of units conducting mathematical operations on their inputs and passing along the results (see Fig 6b). The useful-

ness of an NN lies in its ability to infer a function from observations. In this way, the DL model can approximate the predictor function that describes the observed data which is essen- tial for both classification and regression tasks. The idea of an NN was first proposed by Hebb [48] as an abstraction of a real biological network of neurons in the mammalian brain. Fig 5. Illustration of a Gaussian naïve Bayes classifier. The classifier fits two Gaussian distributions (depicted with contour plots) to the categorized labels in the training set. The decision boundary is determined by the point where the probability densities for the two categories take the same value. https://doi.org/10.1371/journal.pone.0290762.g005 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 11 / 26

A biological NN comprises numerous layers, each consisting of a set of neurons (see Fig 4 in the S1 Appendix). Each neuron consists of dendrites that receive input signal pulses, a cell body that performs a transformation of the incoming combination of pulses, and an axon. The axon carries the output pulse through its trunk (covered with isolating myelin sheaths) to its ramified terminal, which has a set of synapses that connect to the dendrites of the upstream neurons in the network. The functioning of biological NNs has been studied and formalized independently in [49, 50]. Each neuron in the network conducts a non-linear operation on the superposition of the signals received by its dendrites. These signals are weighted by the strength of the connection of its synapses of the upstream neurons. The process carried out by each neuron can therefore be modeled as a simple unit computing the weighted sum of its inputs x1, . . ., xM with weights wj,1, . . ., wj,M, adding a bias term bj and applying a non-linear transformation a(�), called an activation function on the weighted sum (see Fig 4a in the S1 Appendix). This unit, first pro- posed in [51], is referred to as an artificial neuron and constitutes the basic building block of every NN. Information travels through the NN from the input layer towards the output layer, passing through all neurons on its way. For binary classification tasks, the output layer consists of a single neuron that outputs the probability of the observation belonging to one of the classes.

Fig 6. Illustration of a deep-learning model. The classifier provides a complex non-linear function (a) that maps an input feature vector into a predicted label. The function is learned from the training set by adjusting the parameters of a set of internal units arranged in, e.g., a layered fully-connected structure (b). (a) Deep-learning model as a black box. (b) Fully-connected deep neural network. https://doi.org/10.1371/journal.pone.0290762.g006 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 12 / 26

The layers in between are called hidden layers, as they are not directly visible from the outside of the black box. An NN is referred to as a deep NN if it contains two hidden layers or more. Note that NNs could equally well be adapted for regression and multi-class classification tasks. For the latter, the last layer would consist of the same number of neurons as there are classes, each having a softmax activation function and outputting the probability of belonging to that class. It has been proven that sufficiently deep NNs are able to approximate any arbitrarily com- plex function f(�) [52]. Hence, deep NNs are particularly useful in applications where there are large amounts of data and significant computational resources available, and where manual analysis is tedious or the performance of traditional ML algorithms is unsatisfactory. In the context of text classification, NNs were first applied in [53, 54]. Only recently have they started to be used for political science research [55, 56]. There are various neural architectures avail- able for this task. We review the most relevant ones below. Fully-connected neural networks. The first attempt to combine artificial neurons into a layer was made by Rosenblatt [57]. He devised the concept of a perceptron, a binary classifier mapping an input feature vector into one of several available classes. Formally, given a feature vector x, the perceptron outputs ^y ¼ a X M j¼1 wjxj ! ; ð4Þ where fwjgM j¼1 is the set of weights, and a(�) is the activation function. This step is referred to as

the forward pass. Given an actual training output y, the algorithm readjusts its weights accord- ing to the gradient descent rule wj wj þ aðy ^yÞxj; ð5Þ where α is the so-called learning rate which determines the convergence speed of loss minimi- zation. This adjustment process is referred to as the backward pass. The process continues iter- atively until the loss function in Eq (1) is below an acceptance threshold. If the classification problem is linearly separable, the perceptron is guaranteed to converge, and the predictor function is given by the forward pass with the learned weights wj for all j = 1, . . ., M. In a nutshell, the perceptron is an NN with a single layer. To improve its performance, Wer- bos [58] proposed stacking several perceptrons in consecutive layers. That study coined the concept of a multi-layer perceptron, also known as a fully-connected NN (FCNN). FCNNs exhibit a much better learning ability than that of the perceptron, and are the simplest form of a general class of feedforward NNs, which are neural architectures without cycles, being the most common architectures used nowadays. An FCNN comprises a set of fully-connected (or dense) layers stacked in a line topology (see Fig 6b). Due to the presence of multiple layers, an FCNN is capable of having multiple lev- els of abstraction. An example is its use for facial recognition [56]. Based on an input image, the first layers of the NN capture the simple characteristics of the image, such as oriented edges or corners. Then further layers react to more complicated shapes, such as noses or eyes.

Finally, the last layers are able to detect the face itself. In this way, adding extra layers to the NN can enable solutions to a lot of otherwise non-separable problems. For a deep FCNN with P hidden layers, the input-output relation at each layer p = 1, . . ., P + 1 is given in a matrix form as zp ¼ apðWpzp 1 þ bpÞ; ð6Þ PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 13 / 26

where zp−1 and zp are vectors of the inputs and outputs of layer p, respectively. Then Wp is the matrix of weights between the input and output neurons of the layer, while bp is the vector of bias terms of the given layer. Moreover, ap(�) is the activation function, which is often chosen as sigmoid, hyperbolic tangent (tanh), or rectified linear unit (ReLU). Also note that here the input of the first layer is given by the input feature vector z0 = x. Meanwhile, during the train- ing, the output of the last layer in the network is set to the true label, i.e., zP+1 = y. For a new data point, the output of the last layer provides the predicted label, i.e., zPþ1 ¼ ^y. Learning for feedforward NNs is usually done by means of backpropagation. The NN parameters (i.e., weights and biases) for each layer are adjusted based on the adopted loss func- tion which captures the difference (or error) between the observation, y, and the output of the forward pass based on current parameters ^y. The adjustment is done based on the loss function (1). This learning process is often carried out by means of the stochastic gradient descent, or any other known optimization algorithms used for learning, such as, e.g., Adam [59] and RMSProp [60]. This involves recursively propagating the gradients of the parameters back- wards, from the last to the first layer, similarly to Eq (5). It is noteworthy that a mini-batch ver- sion of the gradient update is often used for more robust convergence. More concretely, the

training dataset is split into small batches for which the model loss is calculated and model parameters are updated. In this way, the batch updates are computationally more efficient because they do not need the training data to be held in the computer’s memory. The advantages of FCNNs are their ability to solve complex non-linear problems, efficiency of handling large amounts of data, and quick predictions after the slow process of training is completed. Their disadvantages are slow training, poor interpretability due to their black- box nature, large numbers of parameters due to the fully-connected structure, and ignoring spatial information by accepting only vectorized inputs. For our analysis, we have used the same architecture for all NNs under consideration, with difference in only a single layer that is particular to the given NN. For FCNN, this layer com- prised a fully-connected layer with a certain number of units and a ReLU activation function. The number of neurons was optimized, alongside other hyperparameters, such as batch size, number of epochs, dropout rate, learning rate, regularization strength, embedding dimension, and maximum vocabulary length. Convolutional neural networks. Convolutional neural networks (CNNs) refer to another subclass of feedforward NNs that are designed to capture spatial and temporal dependencies through the application of kernels. CNNs have the benefit of having a reduced number of parameters and reusable weights. The use of CNNs has become a state-of-the-art approach for

the analysis of images [16]. They have also been shown to be useful for text classification [55, 61]. A CNN consists of two parts, where the first part is dedicated to feature extraction, while the second part performs the actual classification by means of an FCNN (see Fig 5 in the S1 Appendix). CNNs often start with an embedding layer that maps a discrete variable to a vector of continuous numbers. This is done by means of a single fully-connected layer (or a set thereof) that is either learned during the training or substituted by a pre-trained embedding. In the subsequent convolution layer, a set of fixed-size kernels slide through the list of embed- dings, performing a convolution operation. This layer functions as the CNN’s feature extractor because it learns to find local spatial features in the output of the embedding layer. The size of the kernel is the number of embeddings it sees at once multiplied by the length of an entire word embedding. The output of this layer is a set of feature maps that serve as the input for the subsequent pooling layer. Zero-padding is often used in this step, surrounding the input so that a feature map does not shrink. PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 14 / 26

The pooling layer then subsamples the feature maps input to it. It does this by selecting a single element from each region of the feature maps covered by its filter. The pooling layer operates on each feature map independently. This reduces the size of the feature representa- tion, thereby effectively reducing the amount of parameters in the network, whilst preserving the most prominent features. Oftentimes, a dropout layer is also applied as a means to improve generalizability. By destroying the learned co-dependencies between neurons that compensate for errors from the previous layers, overfitting is prevented at a cost of longer training. The output is then fed into a fully-connected layer (or an FCNN) that conducts the classification based on the features extracted by the previous layers. CNNs have important advantages, such as automatic feature extraction, with the possibility of using those for new tasks, and weight sharing, which leads to a reduced number of parame- ters. They also have lower computational complexity than FCNNs. Their drawbacks are ignor- ing the position and orientation of the object in their predictions, their need for large amounts of data, and long training times. In our performance comparisons, the CNN architecture includes an embedding layer, a dropout layer, a convolution layer and a max-pooling layer with output flattened to match a single neuron for final classification. We optimize the same hyperparameters as we tuned for the FCNN, except for the number of units. Instead, for a convolution layer we tune the number

of filters and the size of the kernel. Long short-term memory neural networks. An alternative to feedforward NNs for text classification are recurrent neural networks (RNNs). These are NN architectures that are com- prised of a chain of repeating modules. This allows previous outputs to be used as the next inputs of the same neural layer. Because of this feature, RNNs are able to incorporate context from previous steps when dealing with a current step. They can use their internal state—also known as memory—to process sequences of inputs with variable lengths. Unfortunately, plain versions of RNNs suffer from the vanishing gradient problem during training with backpropagation. This limits the number of passes information can make through RNNs, making them impractical. A solution to this problem, so-called long short-term memory (LSTM) NNs, has been proposed [62]. This is nowadays considered as one of the breakthroughs in the field of DL. An LSTM neural network is an RNN with an LSTM cell in place of a neural layer (see Fig 6 in the S1 Appendix). This can let information pass through the cells, as well as adding/remov- ing information while it is passing through a cell. This is regulated by various gates, each com- prising a neural layer and an element-wise multiplication operation. The neural layers are equipped with a sigmoid activation function, which determines the amount of information that should pass through (between 0% and 100%). The forget gate then controls what informa-

tion is kept and what is thrown away from the cell state. It comprises a neural layer with a sig- moid activation. Next, the input gate decides what information to update and what to store. This is followed by a layer with hyperbolic tangent (tanh) activation that outputs a vector of new candidate values that could be added to the cell state to update it. After this, the output gate decides what information to output, which is followed by yet another tanh gate that takes the current state as an input and multiplies the result with the output of the output gate. The benefits of LSTN networks are their ability to handle long-term dependencies, avoid- ance of the vanishing-gradient problem and efficient handling of complex sequential data. The disadvantages are their need for large amounts of data and, consequently, slow training and high computational complexity, as well as their worse performance on data that is not of sequence type or contains a lot of noise. In practice, a bi-directional model [63] is added on top of the LSTM layer. This runs sepa- rately in forward and backward directions, concatenating the predictions of both models, and PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 15 / 26

sending them to an LR model. This allows one, at any point in time, to preserve information from both the past and the future. For our experiments, we optiimze the same parameters as for the FCNN and CNN. There is a slight difference for the bi-directional LSTM layer, where we tune the number of units and the layer dropout rate. Results Next, we evaluate how well the methods discussed in the previous section perform in the task of tweet classification. For the sake of reproducibility, notebooks containing the codes are available at our repository [64]. Data For the analysis, we have collected a dataset [65] with tweets from eight Twitter accounts: @UNOCHA, @UNDP, @FAO, @WHO, @UNICEF, @UNDP, @UNDRR, and @Refugees. As elaborated in the introduction, these accounts correspond to international organizations in the UN system that deal with climate change in different policy areas [25, 26]. The tweets were downloaded and parsed via the Twitter Academic Research API. In total, the dataset contains 222,191 tweets posted by the eight UN organizations from their official accounts. This number represents the total number of tweets by these eight selected UN orga- nizations between the beginning of their tweeting history and the end of 2019. The data com- prises all tweets posted by the official accounts of the selected UN organizations, whether or not these tweets are concerned with climate change. Each tweet is considered as an observation in our dataset. From the entire dataset, 5,750 tweets were randomly selected and labeled manually as either