Short text classification with machine learning in the social sciences: The case of climate change on Twitter

“climate change-related” or “not climate change-related”. Three research assistants, with train- ing in climate governance, each labeled 2,000 tweets, working independently from each other. After this manual coding, we obtained a dataset containing 5,750 observations, see Table 1. We “lost” 250 observations due to overlaps between the three datasets produced by the research assistants, with 125 overlaps between each pair. This overlap between the datasets was used to track the agreement between the three research assistants’ coding. The inter-coder reliability was high: on average, the coders agreed on 96% of coding deci- sions, which corresponds to Cohen’s κ = 0.95 [66], viz., very high inter-coder agreement [67]. Two senior researchers trained the research assistants and supervised the coding. In this man- ual coding, tweets about climate change mitigation and adaptation were both classified as cli- mate change-related, i.e., assigned label “1”. Table 1. Summary of the collected dataset. Organization Account Start date End date Tweets FAO @FAO Jan. 2009 Dec. 2019 753 UNDP @UNDP Jul. 2009 Dec. 2019 1,199 UNDRR @UNDRR Oct. 2010 Dec. 2019 256 UNEP @UNEP May 2009 Dec. 2019 540 UNHCR @Refugees Jun. 2008 Dec. 2019 1,114 UNICEF @UNICEF Jul. 2009 Nov. 2019 910 UNOCHA @UNOCHA Jul. 2011 Jul. 2019 366 WHO @WHO May 2008 Dec. 2019 612 Total 5,750 https://doi.org/10.1371/journal.pone.0290762.t001 PLOS ONE Short text classification with machine learning in the social sciences

PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 16 / 26

Performance assessment For model training and performance assessment, we split the dataset into a training set (85%) and a test set (15%). To avoid data leakage, we made sure that the test set is never used until the final step of the performance evaluation. The latter was carried out by comparing the labels outputted by a classifier to the ground truth (labels marked by our research assistants). Additionally, a part of the training set (ca. 15%) was taken as a validation set for hyperpara- meter tuning. There are many known optimization algorithms used for hyperparameter tuning, e.g., Bayesian optimization [68], random search [69] and metaheuristics, such as genetic algorithm [70], simulated annealing [71], differential evolution [72] and swarm intelligence [73]. For tun- ing the hyperparameters of all the methods considered herein, we have chosen to use Bayesian optimization, i.e., learning the probability distribution of the model loss conditioned on a given hyperparameter from the historical data gathered from its previous iterations. In particu- lar, we used the tree-structured Parzen estimator [74] for our evaluations. For the traditional ML methods, in addition to model-specific parameters, we chose the best feature vectorization approach (count vectorizer vs. TF-IDF). For DL models, the Keras tokenizer is used for the conversion of texts to sequences. We also treated the maximum num- ber of considered features as a hyperparameter and tuned it for all models. All NNs have a common architecture. They contain the following layers: an embedding layer, a dropout layer,

a layer with a certain type of units (FCNN, CNN or LSTM) and a ReLU activation function, a flattening layer (where applicable), and a fully connected layer. The actual weights and biases of the neural network are optimized with the help of the Adam optimizer [59]. The weights and biases of the NNs are initialized using the Xavier rule [75] to avoid problems with vanish- ing gradients. During the training of NNs, we tracked the loss values on the training and validation sets (binary cross-entropy is used as a loss function). We picked the model that achieves the lowest validation loss. We have also used the technique of early stopping to catch the minimum of the validation loss before potential overfitting kicks in. Also, note that in all DL-based methods we utilize self-learned embeddings. An investigation of the improvements from pre-trained embeddings (see, e.g., [76, 77]) has been left for future research. Further details on the NN architectures and optimized hyperparameters can be found in our repository. Finally, it is worth noting that we run all our experiments in the Google Colab environment [78], using graphic processing units (GPUs) to accelerate the training of the DL classifiers. Performance metrics The choice of performance indicators is central to our analysis. For this purpose, it is necessary to consider that our dataset is imbalanced, as only ca. 9% of the tweets in the dataset are about climate change. Prediction accuracy, defined as the share of correct predictions out of all pre-

dictions made, is typically used as a performance metric for classification problems. However, for an imbalanced dataset, accuracy becomes a misleading metric. For instance, for our data- set, a simple guess that a tweet is not about climate change would yield 91% accuracy. How- ever, this is not useful for the classification task at hand. Therefore, when choosing evaluation criteria, one has to take into account the dataset imbalance (see [79] for a discussion). Thus, we relied on alternative performance indicators based on the so-called confusion matrix [79]. For each tweet in the test set, based on the observed features x, the classifier under consideration makes a categorization ^y. It classifies the tweet as either “positive” (about climate change, ^y ¼ 1) or “negative” (not about climate change, ^y ¼ 0). When we compared the pre- dicted label ^y (or a classifier’s decision) with the corresponding manually-coded label y, each PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 17 / 26

observation falls into one of four sets: true negatives (with NTN entries), true positives (with NTP entries), false positives (with NFP entries), false negatives (with NFN entries)—see Fig 7. Based on the confusion matrix, precision refers to the fraction of correctly classified exam- ples among all the examples classified as being in a category. Meanwhile, recall refers to the fraction of correctly classified examples amongst all the examples that belong to a category in the dataset. These two metrics can have values between 0 and 1, and are computed as P ¼ NTP NTP þ NFP ; R ¼ NTP NTP þ NFN : ð7Þ The higher the value of each of these two metrics, the better the performance of the classifier. However, each of these metrics are in a trade-off against the other. Changing the classifier’s cutoff threshold (which determines which of the categories an observation falls into) can improve precision—at the expense of recall. And vice versa, improvements in recall can result in losses in precision. Given this trade-off, the preferred metric in this article is the so-called F1 score as a compro- mise metric [79, 80] which is computed as the harmonic mean of the two aforementioned met- rics, i.e., F1 = 2PR/(P + R). Various alternative metrics have been used in the literature for assessing the performance of classifiers on imbalanced datasets, e.g., ROC AUC [81], PR AUC [82] and MCC [83]. However, all of these come with their respective advantages and disadvan- tages. For the purpose of imbalanced classification, there is no single performance metric that

can capture all aspects and tell us exactly how well a particular classifier performs. It therefore might be more informative to separately report several metrics, including precision and recall [84]. Nonetheless, in this article, we focus on the F1 score as the primary performance metric as it has been widely used in the existing literature to assess the performance of classifiers applied to imbalanced datasets. Performance results Table 2 presents the performance of the ML algorithms under consideration in terms of all of the metrics discussed above given the original collected dataset. The results suggest that the best-performing approaches are the LR and RF classifiers. This finding is remarkable since these classifiers beat all the sophisticated neural classifiers at much lower computational complexity. At the same time, we note that the performance scores are generally quite low (the best method, LR, having F1 � 65% only). This is likely due to the characteristics of the dataset: imbalance and a rather small size. Also notable is the comparatively high performance of the lexicon-based classifier, which reaches F1 � 62%. Another interesting result is the poor perfor- mance of the KNN algorithm (with as little as F1 � 29%). Finally, it is noteworthy that the advanced DL methods (FCNN, CNN and LSTM) require significantly longer training (see Table 3). Fig 7. Confusion matrix containing counts of four sets of test observations. The categorization of each observation is based on the comparison of the predicted label with the true one.

https://doi.org/10.1371/journal.pone.0290762.g007 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 18 / 26

The poor classification performance of all the classifiers discussed above can likely be attrib- uted to class imbalance in the dataset. To investigate the influence of this imbalance, we pre- pare a balanced dataset. To do this, we supplement our original dataset with additional tweets taken from Kaggle’s “Twitter Climate Change Sentiment Dataset” [85]. This dataset contains 43, 943 tweets about climate change, manually labeled for sentiment. We are not interested in the sentiment labels, but instead take all of these tweets as ones about climate change (or labeled with “1”). We pick enough samples from the Kaggle dataset to balance out our own dataset and run the same algorithms on the new balanced dataset. We ensure that the same train/test split (i.e., 85%/15%) is maintained and the training and test sets in the original data- set remain parts of the new training and test sets, without mixing between the two. In this way, we make sure that the models see only new data during the performance evaluation stage. Table 4 shows how the classification algorithms perform on the new, balanced dataset. It is notable how dramatically the overall performance scores have improved when the classifica- tion methods are applied to the balanced dataset. Almost all methods now exhibit very high F1 scores, larger than 96%. The only outliers are the KNN algorithm (F1 � 90%) and the lexicon- based classifier (F1 � 86% only). We note that the poor performance of the KNN classifier, compared to other ML algorithms, is in line with the conclusions of [86]. Meanwhile, the

Table 2. Classification performance of considered classifiers on the original collected dataset. Method Accuracy Precision Recall F1 score AUC ROC AUC PR MCC LR 0.957077 0.829268 0.531250 0.647619 0.913945 0.694443 0.643570 RF 0.952436 0.716981 0.593750 0.649573 0.924048 0.685173 0.627497 Lexicon 0.957077 0.909091 0.468750 0.618557 0.732525 0.709352 0.635333 SVM 0.941995 0.606061 0.625000 0.615385 0.881050 0.641829 0.584106 LSTM 0.944316 0.642857 0.562500 0.600000 0.863565 0.591703 0.571686 FCNN 0.938515 0.593220 0.546875 0.569106 0.890488 0.615837 0.536574 NB 0.937355 0.586207 0.531250 0.557377 0.919447 0.617646 0.524492 CNN 0.935035 0.568966 0.515625 0.540984 0.891115 0.605069 0.506828 KNN 0.938515 1.000000 0.171875 0.293333 0.887287 0.598238 0.401461 Note: The results are sorted by the preferred metric, highlighted in bold font. https://doi.org/10.1371/journal.pone.0290762.t002 Table 3. Training and execution times (in seconds) of considered classifiers on the original collected dataset. Method Training time [s] Execution time [s] NB 0.004 0.008 KNN 0.004 0.156 Lexicon 0.246 0.102 LR 0.255 0.004 SVM 0.485 0.136 RF 0.897 0.160 CNN 119 0.200 FCNN 133 0.319 LSTM 185 1.050 Note: Here, training time is the time required to train an ML model on the training set. Meanwhile, execution time is the time required to run a trained ML model on the test set for the given dataset size and split (85%/15%). https://doi.org/10.1371/journal.pone.0290762.t003 PLOS ONE

Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 19 / 26

relative performance of the lexicon classifier was higher for the imbalanced dataset. This is likely because the proposed keywords better fit the collected dataset of UN organizations, rather than the Kaggle dataset containing all kinds of tweets. In the new setting, the ranking of the best-performing methods has changed. The number one is now CNN with F1 � 97%. This indicates that CNN might be an appropriate choice for classification with balanced datasets when the best possible performance needs to be squeezed out at a cost of higher complexity. In Table 4, CNN is closely followed by the RF and LR classi- fiers. It is however unclear whether the differences in the performance of these classifiers is sta- tistically significant or not. To investigate this, we conduct pair-wise McNemar’s tests [87] on every possible pair of classifiers under consideration. The corresponding methodology is described in the S1 Appendix, where Fig 8 in the S1 Appendix presents the results of the statis- tical tests. The results show that the performance of almost all the considered methods does not significantly differ from one another, with the exception of KNN and lexicon classifiers. That is, for scenarios where the dataset is balanced, it matters less which classification algo- rithm is chosen (with the exception of the KNN and lexicon classifiers). We also sought to investigate the role of the imbalance further. To do this, we sweep the degree of balance in the dataset, adding new data from the Kaggle dataset to the training and

test sets, keeping the same class proportions in both. We then compute the F1 scores for all the considered methods (see Fig 8). The dataset imbalance is captured by the balance ratio, which is defined as the ratio of positive to negative labels in the dataset. As mentioned before, for our original collected dataset, this ratio is only 0.09, hence it is heavily imbalanced. For the new completely balanced dataset, the balance ratio becomes 1.0. For each balance-ratio point in this experiment, we keep the balance between the classes the same for the training, validation and test sets. Fig 8 shows that the performance of all classification algorithms improves with a higher bal- ance ratio. The difference in performance diminishes as the balance ratio increases. This underpins the above observation that for nearly balanced datasets, the classification algorithms tend to perform similarly, with the exception of less capable KNN and lexicon classifiers. At the same time, the same algorithms—namely LR, RF and, to some extent, SVM and LSTM— perform best for almost all degrees of balance. Taken together, we conclude that, for short text classification, it appears to make more sense to turn to traditional ML algorithms instead of computationally-demanding deep neural architectures. The benefits of the latter are marginal, while their training requires significantly Table 4. Classification performance of considered classifiers on the artificially balanced dataset. Method Accuracy Precision Recall F1 score AUC ROC AUC PR MCC

CNN 0.970551 0.977128 0.963659 0.970347 0.990883 0.992459 0.941192 RF 0.970551 0.992136 0.948622 0.969891 0.990770 0.993153 0.942009 LR 0.969298 0.980745 0.957393 0.968928 0.990146 0.992391 0.938863 FCNN 0.968045 0.977011 0.958647 0.967742 0.989988 0.991918 0.936256 SVM 0.968045 0.979461 0.956140 0.967660 0.988149 0.991014 0.936356 LSTM 0.961153 0.956576 0.966165 0.961347 0.986301 0.987585 0.922352 NB 0.960526 0.955390 0.966165 0.960748 0.989931 0.992488 0.921111 KNN 0.895363 0.846323 0.966165 0.902282 0.972934 0.974146 0.798776 Lexicon 0.875313 0.995041 0.754386 0.858161 0.875384 0.936210 0.773593 Note: The results are sorted by the preferred metric, highlighted in bold font. https://doi.org/10.1371/journal.pone.0290762.t004 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 20 / 26

more time. In contrast, DL methods might be more useful when datasets are balanced and of a large scale, and when extensive computational resources are available. Conclusions In this article, we have compared the performance of various methods for the automated cate- gorization of short texts in a small and imbalanced dataset. This classification task frequently occurs in social science research when researchers analyse short texts such as sentences, para- graphs, tweets, or article abstracts. Moreover, it is often the case that the concepts that are being examined are only present in a small proportion of the overall dataset. We have reviewed the most widely used text classifiers and compared their performance in this challenging set- ting. For this purpose, we have used a novel dataset on the communication about climate change on Twitter by eight international organizations operating in the UN system in different policy areas. The article has made three main contributions to social science research engaging in auto- mated text classification. First, it has provided a systematic overview of central text classifica- tion methods, including lexicons, traditional machine learning, and deep learning. In doing so, the article helps to reduce the entry barrier for researchers seeking to use such methods in their work. Computer science is a vast field and it is important to filter for the state-of-the-art methods that can be used to advance various fields in the social sciences. Second, the performance assessment suggests that among the aforementioned classification

methods, lexicons are clearly outperformed by machine-learning methods. However, it is important to take into account the role of class imbalance. For instance, the use of a convolu- tional neural network was found to be the best choice when there is a balanced dataset, but for Fig 8. F1 score vs. balance ratio for the classifiers under consideration. Solid lines show the performance of traditional ML methods, dashed lines show that of DL-based approaches, while the dash-dotted line illustrates the performance of the unsupervised lexicon-based classifier. https://doi.org/10.1371/journal.pone.0290762.g008 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 21 / 26

an imbalanced dataset, its performance is rather mediocre. We therefore recommend that researchers balance their datasets, for instance by complementing the dataset with more labeled data and by applying techniques of active learning [88]. Classifiers trained on balanced datasets are shown to perform much better. Third, we have shown that simple traditional supervised classifiers, such as random forest and logistic regression, tend to perform as well as advanced neural networks in the aforemen- tioned settings, while having much lower computational complexity. It therefore seems war- ranted to use deep learning only in those cases where there is a lot of high-quality data and extensive computational resources. All told, future research should study the application of advanced methods, such as pre- trained embeddings [89] and transformers [90], as well as novel hyperparameter tuning tech- niques [73], in combination with the methods discussed in this article. In the rapidly changing textual data landscape, the need for a systematic understanding of how the prominent methods for coding political texts perform is becoming ever more pressing. Supporting information S1 Appendix. (ZIP) Acknowledgments The authors would like to thank to Hugo de Vos, Nicholas Olczak, and Maryna Povitkina for helpful comments. Earlier version of this paper was presented at the 2021 Annual Meeting of the American Political Science Association. Author Contributions Conceptualization: Karina Shyrokykh, Max Girnyk. Data curation: Karina Shyrokykh, Max Girnyk, Lisa Dellmuth.

Formal analysis: Karina Shyrokykh, Max Girnyk. Funding acquisition: Lisa Dellmuth. Investigation: Karina Shyrokykh, Lisa Dellmuth. Methodology: Karina Shyrokykh, Max Girnyk. Project administration: Lisa Dellmuth. Resources: Lisa Dellmuth. Software: Max Girnyk. Supervision: Karina Shyrokykh, Lisa Dellmuth. Validation: Max Girnyk. Visualization: Max Girnyk. Writing – original draft: Karina Shyrokykh, Max Girnyk, Lisa Dellmuth. Writing – review & editing: Lisa Dellmuth. PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 22 / 26

References 1. Shirky C. Here Comes Everybody: The Power of Organizing without Organizations. Penguin Books; 2008. 2. Cody EM, Reagan AJ, Mitchell L, Dodds PS, Danforth CM. Climate change sentiment on Twitter: An unsolicited public opinion poll. PLoS ONE. 2015; 10(8):e0136092. https://doi.org/10.1371/journal.pone. 0136092 PMID: 26291877 3. Williams HT, McMurray JR, Kurz T, Lambert FH. Network analysis reveals open forums and echo chambers in social media discussions of climate change. Global Environmental Change. 2015; 32:126– 138. https://doi.org/10.1016/j.gloenvcha.2015.03.006 4. Ecker-Ehrhardt M. IO public communication going digital? Understanding social media adoption and use in times of politicization. In: Bjola C, Zaiotti R, editors. Digital Diplomacy and International Organisa- tions Autonomy, Legitimacy and Contestation. 1st ed. Routledge; 2020. p. 21–51. 5. Sebők M, Kacsuk Z. The multiclass classification of newspaper articles with machine learning: The hybrid binary snowball approach. Political Analysis. 2021; 29(2):236–249. https://doi.org/10.1017/pan. 2020.27 6. Hall N. What is adaptation to climate change? Epistemic ambiguity in the climate finance system. International Environmental Agreements. 2017; 17:37–53. https://doi.org/10.1007/s10784-016- 9345-6 7. Persson Å. Global adaptation governance: An emerging but contested domain. WIREs Climate Change. 2019; 10(6):1–18. https://doi.org/10.1002/wcc.618 8. Taboada M, Brooke J, Tofiloski M, Voll K, Stede M. Lexicon-based methods for sentiment analysis.

Computational Linguistics. 2011; 37(2):267–307. https://doi.org/10.1162/COLI_a_00049 9. He Y, Saif H, Wei Z, Wong KF. Quantising opinions for political tweets analysis. In: Proceedings of the 8th International Conference on Language Resources and Evaluation; 2012. p. 3901–3906. 10. Cameletti M, Fabris S, Schlosser S, Toninelli D. Dictionary-based classification of tweets about environ- ment. Journal of Mathematics and Statistical Science. 2020; 8(1). 11. Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. Journal of Machine Learning Research. 2003; 3 (Jan):993–1022. 12. Joachims T. Text categorization with support vector machines: Learning with many relevant features. In: Proceedings of the European Conference on Machine Learning; 1998. p. 137–142. 13. D’Orazio V, Landis ST, Palmer G, Schrodt P. Separating the wheat from the chaff: Applications of auto- mated document classification using support vector machines. Political Analysis. 2014; 22(2):224–242. https://doi.org/10.1093/pan/mpt030 14. He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision; 2015. p. 1026–1034. 15. Popel M, Tomkova M, Tomek J, Kaiser Ł, Uszkoreit J, Bojar O, et al. Transforming machine transla- tion: A deep learning system reaches news translation quality comparable to human professionals. Nature Communications. 2020; 11(1):1–15. https://doi.org/10.1038/s41467-020-18073-9 PMID: 32873773

16. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015; 521(7553):436–444. https://doi.org/10. 1038/nature14539 PMID: 26017442 17. Grimmer J, Stewart BM. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis. 2013; 21(3):267–297. https://doi.org/10.1093/pan/mps028 18. Boussalis C, Coan TG, Holman MR. Climate change communication from cities in the USA. Climatic Change. 2018; 149(2):173–187. https://doi.org/10.1007/s10584-018-2223-1 19. Greene KT, Park B, Colaresi M. Machine learning human rights and wrongs: How the successes and failures of supervised learning algorithms can inform the debate about information effects. Political Anal- ysis. 2019; 27(2):223–230. https://doi.org/10.1017/pan.2018.11 20. Effrosynidis D, Sylaios G, Arampatzis A. Exploring climate change on Twitter using seven aspects: Stance, sentiment, aggressiveness, temperature, gender, topics, and disasters. PLoS ONE. 2022; 17 (9). https://doi.org/10.1371/journal.pone.0274213 PMID: 36129885 21. Falkenberg M, Galeazzi A, Torricelli M, Di Marco N, Larosa F, Sas M, et al. Growing polarization around climate change on social media. Nature Climate Change. 2022; 12:1114–1121. https://doi.org/10.1038/ s41558-022-01527-x 22. Jang SM, Hart PS. Polarized frames on “climate change” and “global warming” across countries and states: Evidence from Twitter big data. Global Environmental Change. 2015; 32:11–17. https://doi.org/ 10.1016/j.gloenvcha.2015.02.010 PLOS ONE Short text classification with machine learning in the social sciences

PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 23 / 26

23. Toupin R, Millerand F, Larivière V. Who tweets climate change papers? Investigating publics of research through users’ descriptions. PLoS ONE. 2022; 17(6):e0268999. https://doi.org/10.1371/ journal.pone.0268999 PMID: 35657791 24. Walter S, Lo¨ rcher I, Bru¨ggemann M. Scientific networks on Twitter: Analyzing scientists’ interactions in the climate change debate. Public Understanding of Science. 2019; 28(6):696–712. https://doi.org/10. 1177/0963662519844131 PMID: 31027461 25. Kural E, Dellmuth L, Gustafsson MT. International organizations and climate change adaptation: A new dataset for the social scientific study of adaptation, 1990–2017. PloS ONE. 2021; 16(9):e0257101. https://doi.org/10.1371/journal.pone.0257101 PMID: 34506547 26. Dellmuth L, Gustafsson MT. Global adaptation governance: How intergovernmental organizations mainstream climate change adaptation. Climate Policy. 2021; 21(7):1–16. https://doi.org/10.1080/ 14693062.2021.1927661 27. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. SciKit-Learn: Machine learning in Python. Journal of Machine Learning Research. 2011; 12:2825–2830. 28. Chollet F, Go¨rner M, Gulli A, Moolayil J, Heaton J, Allaire JJ, et al. Keras; 2015. https://keras.io, accessed Apr. 2023. 29. Verhulst PF. Notice sur la loi que la population suit dans son accroissement. Correspondance Mathe´- matique et Physique. 1838; 10:113–126. 30. Zhang T, Oles FJ. Text categorization based on regularized linear classification methods. Information Retrieval. 2001; 4(1):5–31.

31. Aseervatham S, Gaussier E´ , Antoniadis A, Burlet M, Denneulin Y. Logistic regression and text classifi- cation. In: Gaussier E´ , Yvon F, editors. Textual Information Access: Statistical Model. John Wiley & Sons; 2012. p. 61–84. 32. Cortes C, Vapnik V. Support-vector networks. Machine Learning. 1995; 20(3):273–297. https://doi.org/ 10.1007/BF00994018 33. Scho¨ lkopf B, Smola AJ, Bach F. Learning with Kernels: Support Vector Machines, Regularization, Opti- mization, and Beyond. MIT Press; 2002. 34. Lampert CH. Kernel Methods in Computer Vision. Now Publishers Inc.; 2009. 35. Kwok SW, Carter C. Multiple decision trees. In: Shchter RD, Levitt TS, Kanal LN, Lemmer JF, editors. Machine Intelligence and Pattern Recognition. vol. 9. Elsevier; 1990. p. 327–335. 36. Breiman L. Random forests. Machine Learning. 2001; 45(1):5–32. https://doi.org/10.1023/ A:1010933404324 37. Bouaziz A, Dartigues-Pallez C, da Costa Pereira C, Precioso F, Lloret P. Short text classification using semantic random forest. In: Proceedings of the International Conference on Data Warehousing and Knowledge Discovery; 2014. p. 288–299. 38. Wu Q, Ye Y, Zhang H, Ng MK, Ho SS. ForesTexter: An efficient random forest algorithm for imbalanced text categorization. Knowledge-Based Systems. 2014; 67:105–116. https://doi.org/10.1016/j.knosys. 2014.06.004 39. Quinlan JR. Induction of decision trees. Machine Learning. 1986; 1(1):81–106. https://doi.org/10.1007/ BF00116251 40. Apte´ C, Damerau F, Weiss SM. Automated learning of decision rules for text categorization. ACM

Transactions on Information Systems. 1994; 12(3):233–251. https://doi.org/10.1145/183422.183423 41. Breiman L. Bagging predictors. Machine Learning. 1996; 24(2):123–140. https://doi.org/10.1007/ BF00058655 42. Weiss SM, Kulikowski CA. Computer Systems that Learn: Classification and Prediction Methods from Sta- tistics, Neural Nets, Machine Learning, and Expert Systems. Morgan Kaufmann Publishers Inc.; 1991. 43. Masand B, Linoff G, Waltz D. Classifying news stories using memory based reasoning. In: Proceedings of the 15th Annual International ACM Conference on Research and Development in Information Retrieval; 1992. p. 59–65. 44. Trstenjak B, Mikac S, Donko D. KNN with TF-IDF based framework for text categorization. Procedia Engineering. 2014; 69:1356–1364. https://doi.org/10.1016/j.proeng.2014.03.129 45. Domingos P, Pazzani M. Beyond independence: Conditions for the optimality of the simple Bayesian classifier. In: Proceedings of the 13th International Conference on Machine Learning; 1996. p. 105–112. 46. McCallum A, Nigam K. A comparison of event models for naive bayes text classification. In: Proceed- ings of the AAAI Workshop on Learning for Text Categorization. vol. 752; 1998. p. 41–48. 47. Tang B, Kay S, He H. Toward optimal feature selection in naive Bayes for text categorization. IEEE Transactions on Knowledge and Data Engineering. 2016; 28(9):2508–2521. https://doi.org/10.1109/ TKDE.2016.2563436 PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 24 / 26

48. Hebb DO. The Organization of Behavior. New York: Wiley; 1949. 49. Bain A. Mind and Body: The Theories of their Relation. vol. 4. Appleton; 1873. 50. James W, Burkhardt F, Bowers F, Skrupskelis IK. The Principles of Psychology. vol. 1. Macmillan Lon- don; 1890. 51. McCulloch WS, Pitts W. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics. 1943; 5(4):115–133. https://doi.org/10.1007/BF02478259 52. Cybenko G. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems. 1989; 2(4):303–314. https://doi.org/10.1007/BF02551274 53. Wiener E, Pedersen JO, Weigend AS, et al. A neural network approach to topic spotting. In: Proceed- ings of the 4th Annual Symposium on Document Analysis and Information Retrieval. vol. 317; 1995. p. 332. 54. Ng HT, Goh WB, Low KL. Feature selection, perceptron learning, and a usability case study for text cat- egorization. In: Proceedings of the 20th International ACM Conference on Research and Development in Information Retrieval; 1997. p. 67–73. 55. Kim Y. Convolutional Neural Networks for Sentence Classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing; 2014. p. 1–6. 56. Torres M, Cantu´ F. Learning to see: Convolutional neural networks for the analysis of social science data. Political Analysis. 2021; p. 1–19. 57. Rosenblatt F. The perceptron: A probabilistic model for information storage and organization in the

brain. Psychological review. 1958; 65(6):386. https://doi.org/10.1037/h0042519 PMID: 13602029 58. Werbos P. Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University; 1974. 59. Kingma DP, Ba J. Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations; 2015. p. 1–15. 60. Hinton G, Srivastava N, Swersky K. Coursera course on neural networks for machine learning: Lecture 6a—Overview of mini-batch gradient descent; 2012. 61. Collobert R, Weston J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine learning; 2008. p. 160–167. 62. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation. 1997; 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 PMID: 9377276 63. Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks. 2005; 18(5-6):602–610. https://doi.org/10.1016/j.neunet.2005. 06.042 PMID: 16112549 64. Shyrokykh K. Replication codes and data for: Short Text Classification with Machine Learning in the Social Sciences; 2023. https://github.com/shikarina/short_text_classification. 65. Shyrokykh K, Girnyk M, Dellmuth L. A Dataset of UN Agencies’ Public Communication about Climate Change on Twitter; 2023. https://zenodo.org/record/7633599/#.Y-lbTS8w1qs, accessed Apr. 2023.

66. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960; 20(1):37–46. https://doi.org/10.1177/001316446002000104 67. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; p. 159–174. https://doi.org/10.2307/2529310 PMID: 843571 68. Mockus J. The application of Bayesian methods for seeking the extremum. Towards global optimiza- tion. 1998; 2:117. 69. Bergstra J, Bengio Y. Random search for hyper-parameter optimization. Journal of Machine Learning Research. 2012; 13(2). 70. Aszemi NM, Dominic P. Hyperparameter optimization in convolutional neural network using genetic algorithms. International Journal of Advanced Computer Science and Applications. 2019; 10(6). https:// doi.org/10.14569/IJACSA.2019.0100638 71. Fischetti M, Stringher M. Embedding simulated annealing within stochastic gradient descent. In: Pro- ceedings of the 4th International Conference on Optimization and Learning; 2021. p. 3–13. 72. Schmidt M, Safarani S, Gastinger J, Jacobs T, Nicolas S, Schu¨ lke A. On the performance of differential evolution for hyperparameter tuning. In: Proceedings of the International Joint Conference on Neural Networks; 2019. p. 1–8. 73. Bacanin N, Zivkovic M, Stoean C, Antonijevic M, Janicijevic S, Sarac M, et al. Application of natural lan- guage processing and machine learning boosted with Swarm Intelligence for spam email filtering. Math- ematics. 2022; 10(22):4173. https://doi.org/10.3390/math10224173 PLOS ONE

Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 25 / 26

74. Bergstra J, Bardenet R, Bengio Y, Ke´ gl B. Algorithms for hyper-parameter optimization. Advances in Neural Information Processing Systems. 2011; 24. 75. Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Pro- ceedings of the 13th International Conference on Artificial Intelligence and Statistics; 2010. p. 249–256. 76. Jang B, Kim I, Kim JW. Word2vec convolutional neural networks for classification of news articles and tweets. PLoS ONE. 2019; 14(8):e0220976. https://doi.org/10.1371/journal.pone.0220976 PMID: 31437181 77. Nandanwar AK, Choudhary J. Semantic features with contextual knowledge-based web page categori- zation using the GloVe model and stacked BiLSTM. Symmetry. 2021; 13(10):1772. https://doi.org/10. 3390/sym13101772 78. Google. Colaboratory; 2023. https://research.google.com/colaboratory, verified Apr. 2023. 79. He H, Garcia EA. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engi- neering. 2009; 21(9):1263–1284. https://doi.org/10.1109/TKDE.2008.239 80. Denny MJ, Spirling A. Text preprocessing for unsupervised learning: Why it matters, when it misleads, and what to do about it. Political Analysis. 2018; 26(2):168–189. https://doi.org/10.1017/pan.2017.44 81. Fawcett T. ROC graphs: Notes and practical considerations for researchers. Machine Learning. 2004; 31(1):1–38. 82. Davis J, Goadrich M. The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine learning; 2006. p. 233–240.

guage understanding. In: Proceedings of the Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies; 2019. p. 4171–4186. PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 26 / 26