Short text classification with machine learning in the social sciences: The case of climate change on Twitter

sending them to an LR model. This allows one, at any point in time, to preserve information from both the past and the future. For our experiments, we optiimze the same parameters as for the FCNN and CNN. There is a slight difference for the bi-directional LSTM layer, where we tune the number of units and the layer dropout rate. Results Next, we evaluate how well the methods discussed in the previous section perform in the task of tweet classification. For the sake of reproducibility, notebooks containing the codes are available at our repository [64]. Data For the analysis, we have collected a dataset [65] with tweets from eight Twitter accounts: @UNOCHA, @UNDP, @FAO, @WHO, @UNICEF, @UNDP, @UNDRR, and @Refugees. As elaborated in the introduction, these accounts correspond to international organizations in the UN system that deal with climate change in different policy areas [25, 26]. The tweets were downloaded and parsed via the Twitter Academic Research API. In total, the dataset contains 222,191 tweets posted by the eight UN organizations from their official accounts. This number represents the total number of tweets by these eight selected UN orga- nizations between the beginning of their tweeting history and the end of 2019. The data com- prises all tweets posted by the official accounts of the selected UN organizations, whether or not these tweets are concerned with climate change. Each tweet is considered as an observation in our dataset. From the entire dataset, 5,750 tweets were randomly selected and labeled manually as either

“climate change-related” or “not climate change-related”. Three research assistants, with train- ing in climate governance, each labeled 2,000 tweets, working independently from each other. After this manual coding, we obtained a dataset containing 5,750 observations, see Table 1. We “lost” 250 observations due to overlaps between the three datasets produced by the research assistants, with 125 overlaps between each pair. This overlap between the datasets was used to track the agreement between the three research assistants’ coding. The inter-coder reliability was high: on average, the coders agreed on 96% of coding deci- sions, which corresponds to Cohen’s κ = 0.95 [66], viz., very high inter-coder agreement [67]. Two senior researchers trained the research assistants and supervised the coding. In this man- ual coding, tweets about climate change mitigation and adaptation were both classified as cli- mate change-related, i.e., assigned label “1”. Table 1. Summary of the collected dataset. Organization Account Start date End date Tweets FAO @FAO Jan. 2009 Dec. 2019 753 UNDP @UNDP Jul. 2009 Dec. 2019 1,199 UNDRR @UNDRR Oct. 2010 Dec. 2019 256 UNEP @UNEP May 2009 Dec. 2019 540 UNHCR @Refugees Jun. 2008 Dec. 2019 1,114 UNICEF @UNICEF Jul. 2009 Nov. 2019 910 UNOCHA @UNOCHA Jul. 2011 Jul. 2019 366 WHO @WHO May 2008 Dec. 2019 612 Total 5,750 https://doi.org/10.1371/journal.pone.0290762.t001 PLOS ONE Short text classification with machine learning in the social sciences

PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 16 / 26