Short text classification with machine learning in the social sciences: The case of climate change on Twitter

Performance assessment For model training and performance assessment, we split the dataset into a training set (85%) and a test set (15%). To avoid data leakage, we made sure that the test set is never used until the final step of the performance evaluation. The latter was carried out by comparing the labels outputted by a classifier to the ground truth (labels marked by our research assistants). Additionally, a part of the training set (ca. 15%) was taken as a validation set for hyperpara- meter tuning. There are many known optimization algorithms used for hyperparameter tuning, e.g., Bayesian optimization [68], random search [69] and metaheuristics, such as genetic algorithm [70], simulated annealing [71], differential evolution [72] and swarm intelligence [73]. For tun- ing the hyperparameters of all the methods considered herein, we have chosen to use Bayesian optimization, i.e., learning the probability distribution of the model loss conditioned on a given hyperparameter from the historical data gathered from its previous iterations. In particu- lar, we used the tree-structured Parzen estimator [74] for our evaluations. For the traditional ML methods, in addition to model-specific parameters, we chose the best feature vectorization approach (count vectorizer vs. TF-IDF). For DL models, the Keras tokenizer is used for the conversion of texts to sequences. We also treated the maximum num- ber of considered features as a hyperparameter and tuned it for all models. All NNs have a common architecture. They contain the following layers: an embedding layer, a dropout layer,

a layer with a certain type of units (FCNN, CNN or LSTM) and a ReLU activation function, a flattening layer (where applicable), and a fully connected layer. The actual weights and biases of the neural network are optimized with the help of the Adam optimizer [59]. The weights and biases of the NNs are initialized using the Xavier rule [75] to avoid problems with vanish- ing gradients. During the training of NNs, we tracked the loss values on the training and validation sets (binary cross-entropy is used as a loss function). We picked the model that achieves the lowest validation loss. We have also used the technique of early stopping to catch the minimum of the validation loss before potential overfitting kicks in. Also, note that in all DL-based methods we utilize self-learned embeddings. An investigation of the improvements from pre-trained embeddings (see, e.g., [76, 77]) has been left for future research. Further details on the NN architectures and optimized hyperparameters can be found in our repository. Finally, it is worth noting that we run all our experiments in the Google Colab environment [78], using graphic processing units (GPUs) to accelerate the training of the DL classifiers. Performance metrics The choice of performance indicators is central to our analysis. For this purpose, it is necessary to consider that our dataset is imbalanced, as only ca. 9% of the tweets in the dataset are about climate change. Prediction accuracy, defined as the share of correct predictions out of all pre-

dictions made, is typically used as a performance metric for classification problems. However, for an imbalanced dataset, accuracy becomes a misleading metric. For instance, for our data- set, a simple guess that a tweet is not about climate change would yield 91% accuracy. How- ever, this is not useful for the classification task at hand. Therefore, when choosing evaluation criteria, one has to take into account the dataset imbalance (see [79] for a discussion). Thus, we relied on alternative performance indicators based on the so-called confusion matrix [79]. For each tweet in the test set, based on the observed features x, the classifier under consideration makes a categorization ^y. It classifies the tweet as either “positive” (about climate change, ^y ¼ 1) or “negative” (not about climate change, ^y ¼ 0). When we compared the pre- dicted label ^y (or a classifier’s decision) with the corresponding manually-coded label y, each PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 17 / 26