Short text classification with machine learning in the social sciences: The case of climate change on Twitter

the broader public. Social media has thus been defined as a social tool shaping new and unmanaged communication dynamics [1]. For example, the microblogging site Twitter is often used to package information in short messages [2–4]. In the study of short texts, researchers are often interested in categorizing or labelling texts according to their topic, relevance, or stance before carrying out further analysis. This text clas- sification process is a key challenge for two main reasons. First, manual classification of large numbers of texts might not be feasible and automated methods might be required. However, little is known about the relative performance of such methods in the context of social science research [5]. Second, concepts in the social sciences are often ambiguous and complex. This makes it difficult for automated classifiers to categorize texts based on those. A case in point is the topic of climate change and the risks it causes for societies and ecosys- tems. Over the past thirty years, framings of climate change as a policy challenge have prolifer- ated and differentiated across sectors and scales. Political debates about “climate adaptation”, “climate mitigation”, “carbon emissions”, “climate-induced displacement”, “disaster risk reduction”, and “climate vulnerability” all illustrate the complex nature of the concept [6, 7]. When classifying texts about climate change and its governance, researchers need to know how the concept of climate change is being referred to in the specific context in which it is being studied.

Several approaches to automated text classification are known. Lexicon-based methods are the most popular among social scientists [8–10]. These are based on lexicons or dictionaries of meaningful words related to a topic of interest, crafted by experts, and made available to the research community. Because such dictionaries are not always available, machine-learning (ML) algorithms are being increasingly applied. The two main categories of ML methods are supervised and unsupervised learning. In unsupervised ML methods, the computer may cluster the words in texts around unknown categories. These clusters, and the categories they might indicate, are then inter- preted by the researcher [11]. However, such approaches might not be a good choice when seeking to classify texts using labels that have been defined a priori. When the categories are known, supervised learning is more appropriate [12, 13]. Here, researchers first manually label a smaller dataset, which is then used to train a model that identifies the specified categories in a larger dataset in an automated way. Remarkably, in recent years, a subset of these methods, deep learning (DL), has achieved human-level performance carrying out certain tasks, such as image recognition [14] and text translation [15]. DL methods have proven very efficient at handling complex non-linear relationships within data, but require large amounts of advanced computations [16]. The choice of algorithm used for ML depends principally on the task at hand, but should

also be motivated by model performance. In this article, we compare several alternatives to find the best-performing algorithm. Drawing on the literature on text classification in the social sciences [5, 9, 17–19], we identify the most widely-used lexicon-based and supervised ML text-classification methods. We then investigate the performance of these methods in a typical social science research setting: a small labeled dataset with rare-event data. Our empiri- cal focus is on Twitter communication about climate change, which is studied in a rapidly growing body of interdisciplinary social science research [2, 20–24]. To delineate the empirical material, we make use of the Twitter communication of eight international organizations in different policy areas that are known to be central in communi- cating about climate change, and which are comparable in their communication as they are all part of the United Nations (UN): the Food and Agriculture Organization (FAO), the Office for the Coordination of Humanitarian Affairs (UNOCHA), the UN Development Programme (UNDP), the UN Office for Disaster Risk Reduction (UNDRR), the UN Environmental PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 2 / 26 Funding: This work was supported by the project: “Glocalizing Climate Governance: The role of Integrated Governance for a Just and Legitimate Adaptation to Climate Risks (GlocalClim)” funded by Formas under grant number 2018-01705 (Svenska Forskningsrådet Formas, Grant/Award

Number: 2018-01705). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist.