Short text classification with machine learning in the social sciences: The case of climate change on Twitter

becomes highly misleading and does not reflect its out-of-sample performance, which is known as overfitting. Thus, a general rule for picking the model is the following: the model should be complex enough to overcome underfitting, but simple enough to not reach overfitting. To detect overfitting, the entire dataset D can be split into two parts: a training set Dtrain and a validation set Dvalid. This makes it possible to validate the out-of-sample performance of a model on the validation set. Then several models can be fitted to the training set, while also val- idating their out-of-sample performance using the validation set. This validation can be used as a basis for selecting a model that is complex enough to generalize to new examples and per- forms the best in terms of average loss among all other considered models. Following this, the best model can be retrained using the entire dataset (minus the test set), whilst also ensuring that it generalizes well when applied to as-yet-unseen data. This approach makes it possible to validate the models and select the best one in terms of the out-of-sample loss. However, it might still not be possible to assess the actual performance of the model on as yet unseen data. This is because the process of model selection contaminates the dataset, injecting a dependency between the model and the data based on which it was selected. Hence, the validation loss is not representative of the actual performance on yet unseen data. To tackle this problem, it is necessary to make yet another partition of the dataset

D. That is, prior to cutting subsets Dtrain and Dvalid from the dataset D, another subset of data, called a test set Dtest is withheld from the validation set. This test set is not touched during the training and validation, and is used purely for assessing the classifier’s performance on unseen data. Traditional machine-learning classifiers By traditional ML classifiers we mean a collection of ML algorithms which have previously been developed for classification tasks. Those are built on different principles and have differ- ent computational complexities. Their performance depends on the task at hand. In this sec- tion, we will review the most popular ML classifiers and specify the variants of the classifiers that will be used for our subsequent performance comparisons. In our analysis, we use the Sci- Kit-Learn implementations [27] of the traditional ML algorithms, whereas for the deep neural classifiers, we use the TensorFlow implementations set up via the Keras API [28]. For each of the methods, we test the three available vectorizers, as well as optimize the number of features to maximize the classifier’s performance. Logistic regression. Logistic regression (LR) is a classification algorithm that was invented in [29] and has been widely applied in text classification [30, 31]. The classifier mod- els a binary random variable representing the label y given the data x. The model, parameter- ized by weights w, is given by ^f wðxÞ ¼ s w0 þ X M j¼1 wjxj ! ; ð2Þ where σ(�) is the sigmoid function representing the conditional probability of a positive class

label given the observed features x1, . . ., xM. An example of a LR classifier in action is shown in Fig 1 on exemplary synthetic data. The two classes are separated by a decision boundary, which is determined by the equal-probability line on the sigmoid-based probability map (see the black solid line on Fig 1). For all data points with probability above the decision threshold we predict ^y ¼ 1, while for the rest we predict ^y ¼ 0. PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 5 / 26