Short text classification with machine learning in the social sciences: The case of climate change on Twitter

an imbalanced dataset, its performance is rather mediocre. We therefore recommend that researchers balance their datasets, for instance by complementing the dataset with more labeled data and by applying techniques of active learning [88]. Classifiers trained on balanced datasets are shown to perform much better. Third, we have shown that simple traditional supervised classifiers, such as random forest and logistic regression, tend to perform as well as advanced neural networks in the aforemen- tioned settings, while having much lower computational complexity. It therefore seems war- ranted to use deep learning only in those cases where there is a lot of high-quality data and extensive computational resources. All told, future research should study the application of advanced methods, such as pre- trained embeddings [89] and transformers [90], as well as novel hyperparameter tuning tech- niques [73], in combination with the methods discussed in this article. In the rapidly changing textual data landscape, the need for a systematic understanding of how the prominent methods for coding political texts perform is becoming ever more pressing. Supporting information S1 Appendix. (ZIP) Acknowledgments The authors would like to thank to Hugo de Vos, Nicholas Olczak, and Maryna Povitkina for helpful comments. Earlier version of this paper was presented at the 2021 Annual Meeting of the American Political Science Association. Author Contributions Conceptualization: Karina Shyrokykh, Max Girnyk. Data curation: Karina Shyrokykh, Max Girnyk, Lisa Dellmuth.

Formal analysis: Karina Shyrokykh, Max Girnyk. Funding acquisition: Lisa Dellmuth. Investigation: Karina Shyrokykh, Lisa Dellmuth. Methodology: Karina Shyrokykh, Max Girnyk. Project administration: Lisa Dellmuth. Resources: Lisa Dellmuth. Software: Max Girnyk. Supervision: Karina Shyrokykh, Lisa Dellmuth. Validation: Max Girnyk. Visualization: Max Girnyk. Writing – original draft: Karina Shyrokykh, Max Girnyk, Lisa Dellmuth. Writing – review & editing: Lisa Dellmuth. PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 22 / 26