Learning from climate change news: Is the world on the same page?

text.TfidfVectorizer) from Scikit Learn. The n-gram ranges could be of different lengths, so that not only word unigrams, but also bi- and trigrams, as well as character multigrams could be included by choice during experiments. The headline and the body text of each article were handled separately, allowing the use of different settings for each element. Classification Architecture. Classification was implemented using Scikit Learn’s LinearSVC (scikit- learn.org/stable/modules/generated/sklearn.svm.LinearSVC) class. This features a linear implementation of support vector machines (scikit-learn.org/stable/modules/svm) (SVM), which is a popular and effective architecture in natural language processing. The default set- tings for LinearSVC were used so that the focus could be on feature generation. For each of the three labels (countries, political orientations and COP meetings) an inde- pendent classification model was trained. All three models were trained consecutively during experiments with different features and settings. This allowed to constantly evaluate the impact of changes across aspects. During experiments, the models were trained in a single run on the Train data and consec- utively evaluated on the Dev set. Eventually, a final model was trained using the optimal con- figuration on the Train-Dev data and evaluated on the so far unseen Test set. Performance of each created model was evaluated by means of Precision, Recall and F1-score. The goal was to determine and analyze the most informative features, i.e. the textual fea-

tures that contribute most to the model’s decision (as done before, for example, by [12, 13]). This is considered an objective way to determine which terms are characteristic or rather uncharacteristic for each country, political orientation and COP meeting. In a broader sense, this allows to uncover topical and terminological differences between the articles for these labels. While this analysis is purely textual and does not consider context or meaning, it can still provide insight into a large dataset in a neutral manner. Concretely, in order to assess the contribution of each single feature, the model’s coefficients were extracted. In Scikit Learn, coefficents can be easily accessed using LinearSVC.coef_. A more detailed explanation can be found on scikit-learn.org/stable/modules/svm.html. Both positive and negative coefficients are generated for each class. These are the weights of the fea- tures, with absolutely higher weights indicating a stronger contribution of the corresponding feature to the models’ decision. Since a lot of features were generated, only the ones with the most positive and negative coefficients were presented. For each given class (for example the country “Australia” in coun- try detection), features with positive weights can be seen as strongly characteristic for that class, and ones with negative weights as uncharacteristic. This allowed to analyze the textual characteristics of each class. To provide more perspective for the scores that would be yielded by the SVM, a baseline

was first generated using the DummyClassifier (scikit-learn.org/stable/modules/ generated/sklearn.dummy.DummyClassifier) class. A stratified strategy was employed, which leads to random predictions solely based on the class distribution within the training data. This can be interpreted as educated guessing. The SVM models should thus at least outperform this baseline to be considered useful. When trained on the Train-Dev set and evaluated on the Test data, the baseline yields a macro F1-score of 0.33 for countries, 0.51 for political orientations and 0.03 for COP meetings. These scores are as expected for random guessing with 3, 2 and 26 classes respectively. Experiments. Several experiments were conducted to determine the most useful combi- nation of features and preprocessing steps. This assessment was based both on performance PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 11 / 27