Short text classification with machine learning in the social sciences: The case of climate change on Twitter

classified as not dealing with climate change. The number of tokenized words which need to match for positive classification is a design parameter. There are many possible ways to classify tweets with respect to the topic of climate change. However, there is no standard lexicon for this type of classification. For this article, in order to benchmark the ML methods with a reasonable classifier, we select the lexicon from [3]. It con- sists of the following key terms: “climate”, “climatechange”, “globalwarming”, “climaterealists” and “agw”(an abbreviation for “anthropogenic global warming”). The lexicon has a decent per- formance and constitutes a good baseline for our empirical performance assessment. Supervised approach to text classification An efficient alternative to lexicon-based classifiers is the use of supervised ML methods. This class of methods requires a manually labeled dataset D (see Eq (1) and Fig 3 in the S1 Appen- dix). By fitting a set of N examples given in this labeled dataset, the methods obtain a model that approximates the predictor function. To fit the model, we choose a mapping ^f ð�Þ that minimizes the empirical average loss. This means that, in fitting the model, the computer attempts to minimize the difference between the predicted value and the actual observation. This can be formally expressed as: Lð^f Þ ¼ 1 N X N i¼1 ‘ ^f ðxðiÞÞ; yðiÞ � � ; ð1Þ where ‘ð^f ðxðiÞÞ; yðiÞÞ is some chosen loss function that reflects the closeness of the model pre- diction ^f ðxðiÞÞ for features x(i) to the true label y(i) for observation i.

The model ^f ð�Þ, being a function of the features x, is also a function of a set of its own parameters w = [w0, . . ., wK−1]. The fitting of the model is therefore carried out by searching the parameter vector w that minimizes the average loss in Eq (1). This process is referred to as the training of the model. By monitoring loss Lð^f Þ we can track the in-sample performance of the model, or more specifically how closely ^f wð�Þ corresponds with the data to which it is being fitted. Because the model learns the “average” mapping from features into labels, we also expect the model to have a decent out-of-sample performance (i.e., performance on new and as-yet- unseen examples). Unfortunately, the in-sample performance of a model might not be indica- tive of its out-of-sample performance, as we shall see below. If the model is too simple (i.e., it has too few parameters w to capture the behavior of the data), it is unable to fit the data well and has poor in-sample performance. This is known as underfitting and is an indicator that the model is poor in general. To avoid underfitting, more complex models have to be considered. The true predictor f(�) can be a complicated relation between features and labels. To approximate it well, the model ^f wð�Þ may sometimes need to be complicated and include many parameters K. For example, a linear model of this kind could be given by the following expression: ^f wðxÞ ¼ w0 þ w1φ1ðxÞ þ � � � þ wK 1φK 1ðxÞ, where φk(x) is some function of features x for k = 1, . . ., K − 1. More complicated models (e.g.,

polynomials) might involve even larger numbers of parameters. The more complex the model ^f wð�Þ, the greater the possibility to decrease the empirical loss Lð^f Þ in Eq (1) and improve its in-sample performance on the given dataset D. However, when the number of parameters becomes too high (relative to the number of examples N in the data- set), the training process might start confusing the noise in the data with the behavior of the feature variables. The model ^f wð�Þ becomes tied to particular examples in the dataset and does not generalize to unseen data. That is, the in-sample performance of the model on the dataset PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 4 / 26