Short text classification with machine learning in the social sciences: The case of climate change on Twitter

observation falls into one of four sets: true negatives (with NTN entries), true positives (with NTP entries), false positives (with NFP entries), false negatives (with NFN entries)—see Fig 7. Based on the confusion matrix, precision refers to the fraction of correctly classified exam- ples among all the examples classified as being in a category. Meanwhile, recall refers to the fraction of correctly classified examples amongst all the examples that belong to a category in the dataset. These two metrics can have values between 0 and 1, and are computed as P ¼ NTP NTP þ NFP ; R ¼ NTP NTP þ NFN : ð7Þ The higher the value of each of these two metrics, the better the performance of the classifier. However, each of these metrics are in a trade-off against the other. Changing the classifier’s cutoff threshold (which determines which of the categories an observation falls into) can improve precision—at the expense of recall. And vice versa, improvements in recall can result in losses in precision. Given this trade-off, the preferred metric in this article is the so-called F1 score as a compro- mise metric [79, 80] which is computed as the harmonic mean of the two aforementioned met- rics, i.e., F1 = 2PR/(P + R). Various alternative metrics have been used in the literature for assessing the performance of classifiers on imbalanced datasets, e.g., ROC AUC [81], PR AUC [82] and MCC [83]. However, all of these come with their respective advantages and disadvan- tages. For the purpose of imbalanced classification, there is no single performance metric that

can capture all aspects and tell us exactly how well a particular classifier performs. It therefore might be more informative to separately report several metrics, including precision and recall [84]. Nonetheless, in this article, we focus on the F1 score as the primary performance metric as it has been widely used in the existing literature to assess the performance of classifiers applied to imbalanced datasets. Performance results Table 2 presents the performance of the ML algorithms under consideration in terms of all of the metrics discussed above given the original collected dataset. The results suggest that the best-performing approaches are the LR and RF classifiers. This finding is remarkable since these classifiers beat all the sophisticated neural classifiers at much lower computational complexity. At the same time, we note that the performance scores are generally quite low (the best method, LR, having F1 � 65% only). This is likely due to the characteristics of the dataset: imbalance and a rather small size. Also notable is the comparatively high performance of the lexicon-based classifier, which reaches F1 � 62%. Another interesting result is the poor perfor- mance of the KNN algorithm (with as little as F1 � 29%). Finally, it is noteworthy that the advanced DL methods (FCNN, CNN and LSTM) require significantly longer training (see Table 3). Fig 7. Confusion matrix containing counts of four sets of test observations. The categorization of each observation is based on the comparison of the predicted label with the true one.

https://doi.org/10.1371/journal.pone.0290762.g007 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 18 / 26