Short text classification with machine learning in the social sciences: The case of climate change on Twitter

RESEARCH ARTICLE Short text classification with machine learning in the social sciences: The case of climate change on Twitter Karina ShyrokykhID*, Max Girnyk, Lisa DellmuthID Department of Economic History and International Relations, Stockholm University, Stockholm, Sweden * karina.shyrokykh@ekohist.su.se Abstract To analyse large numbers of texts, social science researchers are increasingly confronting the challenge of text classification. When manual labeling is not possible and researchers have to find automatized ways to classify texts, computer science provides a useful toolbox of machine-learning methods whose performance remains understudied in the social sciences. In this article, we compare the performance of the most widely used text classifiers by applying them to a typical research scenario in social science research: a rela- tively small labeled dataset with infrequent occurrence of categories of interest, which is a part of a large unlabeled dataset. As an example case, we look at Twitter communication regarding climate change, a topic of increasing scholarly interest in interdisciplinary social science research. Using a novel dataset including 5,750 tweets from various international organizations regarding the highly ambiguous concept of climate change, we evaluate the performance of methods in automatically classifying tweets based on whether they are about climate change or not. In this context, we highlight two main findings. First, supervised machine-learning methods perform better than state-of-the-art lexicons, in particular as

class balance increases. Second, traditional machine-learning methods, such as logistic regression and random forest, perform similarly to sophisticated deep-learning methods, whilst requiring much less training time and computational resources. The results have important implications for the analysis of short texts in social science research. Introduction Contemporary society is characterized by the ever-greater production of textual data. Citizens post on social media, politicians give speeches which are commented upon live, and interna- tional organizations publish reports. Such activities provide social scientists with large amounts of freely available data, enabling them to advance theories in areas such as interna- tional relations, comparative politics, media studies, and public opinion. An example is the field of social media studies which is growing apace, as social networks have become an important aspect of people’s lives. The popularity of social media platforms means that they often serve as a means of communication for citizens living in distant places, civil society organizations, business representatives, and political institutions. Such societal and political actors tend to make use of social media platforms to exchange information with PLOS ONE PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 1 / 26 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Shyrokykh K, Girnyk M, Dellmuth L (2023) Short text classification with machine learning in the social sciences: The case of climate

change on Twitter. PLoS ONE 18(9): e0290762. https://doi.org/10.1371/journal.pone.0290762 Editor: Nebojsa Bacanin, Univerzitet Singidunum, SERBIA Received: December 30, 2022 Accepted: August 15, 2023 Published: September 29, 2023 Peer Review History: PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0290762 Copyright: © 2023 Shyrokykh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: The data underlying the results presented in the study are available here: https://zenodo.org/record/7633599#.Y- lbTS8w1qs.