Short text classification with machine learning in the social sciences: The case of climate change on Twitter

relative performance of the lexicon classifier was higher for the imbalanced dataset. This is likely because the proposed keywords better fit the collected dataset of UN organizations, rather than the Kaggle dataset containing all kinds of tweets. In the new setting, the ranking of the best-performing methods has changed. The number one is now CNN with F1 � 97%. This indicates that CNN might be an appropriate choice for classification with balanced datasets when the best possible performance needs to be squeezed out at a cost of higher complexity. In Table 4, CNN is closely followed by the RF and LR classi- fiers. It is however unclear whether the differences in the performance of these classifiers is sta- tistically significant or not. To investigate this, we conduct pair-wise McNemar’s tests [87] on every possible pair of classifiers under consideration. The corresponding methodology is described in the S1 Appendix, where Fig 8 in the S1 Appendix presents the results of the statis- tical tests. The results show that the performance of almost all the considered methods does not significantly differ from one another, with the exception of KNN and lexicon classifiers. That is, for scenarios where the dataset is balanced, it matters less which classification algo- rithm is chosen (with the exception of the KNN and lexicon classifiers). We also sought to investigate the role of the imbalance further. To do this, we sweep the degree of balance in the dataset, adding new data from the Kaggle dataset to the training and

test sets, keeping the same class proportions in both. We then compute the F1 scores for all the considered methods (see Fig 8). The dataset imbalance is captured by the balance ratio, which is defined as the ratio of positive to negative labels in the dataset. As mentioned before, for our original collected dataset, this ratio is only 0.09, hence it is heavily imbalanced. For the new completely balanced dataset, the balance ratio becomes 1.0. For each balance-ratio point in this experiment, we keep the balance between the classes the same for the training, validation and test sets. Fig 8 shows that the performance of all classification algorithms improves with a higher bal- ance ratio. The difference in performance diminishes as the balance ratio increases. This underpins the above observation that for nearly balanced datasets, the classification algorithms tend to perform similarly, with the exception of less capable KNN and lexicon classifiers. At the same time, the same algorithms—namely LR, RF and, to some extent, SVM and LSTM— perform best for almost all degrees of balance. Taken together, we conclude that, for short text classification, it appears to make more sense to turn to traditional ML algorithms instead of computationally-demanding deep neural architectures. The benefits of the latter are marginal, while their training requires significantly Table 4. Classification performance of considered classifiers on the artificially balanced dataset. Method Accuracy Precision Recall F1 score AUC ROC AUC PR MCC

CNN 0.970551 0.977128 0.963659 0.970347 0.990883 0.992459 0.941192 RF 0.970551 0.992136 0.948622 0.969891 0.990770 0.993153 0.942009 LR 0.969298 0.980745 0.957393 0.968928 0.990146 0.992391 0.938863 FCNN 0.968045 0.977011 0.958647 0.967742 0.989988 0.991918 0.936256 SVM 0.968045 0.979461 0.956140 0.967660 0.988149 0.991014 0.936356 LSTM 0.961153 0.956576 0.966165 0.961347 0.986301 0.987585 0.922352 NB 0.960526 0.955390 0.966165 0.960748 0.989931 0.992488 0.921111 KNN 0.895363 0.846323 0.966165 0.902282 0.972934 0.974146 0.798776 Lexicon 0.875313 0.995041 0.754386 0.858161 0.875384 0.936210 0.773593 Note: The results are sorted by the preferred metric, highlighted in bold font. https://doi.org/10.1371/journal.pone.0290762.t004 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 20 / 26