Learning from climate change news: Is the world on the same page?

Countries The model to distinguish countries scored an overall F1-score of 0.90. This was considered a trustworthy performance, thus feature analysis could proceed. Among these features, some keywords covered climate change directly. These are useful in answering the tertiary research question on countries (“What major topical and terminological differences can be found between countries and how could they be explained?”). For structure, four categories were defined, being the mere terminology to describe climate change, its impact, the causes and taken measures. It seems likely that these differences are meaningful and interesting. • Terminology Climate change may be a global problem, but it is not necessarily a global term. It turns out that the three countries use different terms to refer to the concept. This is in line with the findings from Grundmann and Scott [15]. Whereas climate is a characteristic term for the UK, greenhouse is more common in Australia. This can be augmented as greenhouse emis- sions, while this seems to be carbon emissions in the UK. In Canada, environmental is used, which is less specific. UK: climate, carbon emissions CA: environmental AU: greenhouse, greenhouse emissions • Effects The countries also describe different words that refer to the effects of climate change. While Australia seems to be worried about bushfires and drought, the UK takes on deforestation and wildlife. It appears Australia also considers longer-term effects such as asylum seekers and a

global financial crisis. At the same time, vacation could suggest what the Canadians fear. UK: deforestation, wildlife CA: vacation AU: bushfires, asylum, asylum seekers, global financial crisis, drought • Causes Various causes of climate change also came up. Australia speaks about mining, while Canada mentions oil sands and natural gas. The UK brings up flights. Canada also seems to link glob- alization to the topic. Australia is recognized by the absence of snow and oil, which therefore seem characteristic for the other countries. UK: flights CA: oil sands, natural gas, globalization, pipeline AU: mining UK/CA: oil, snow • Measures Lastly, there are different measures against climate change that characterize the countries. The UK speaks about low carbon, apparently leaving the means by which that must be achieved in the middle. Australia mentions uranium, a carbon-free form of energy produc- tion. This is likely related to Australia having the world’s largest resource of uranium, and being one of the largest exporters of uranium worldwide (world-nuclear.org/information- library/country-profiles/countries-a-f/australia.aspx). This shows that characteristics of coun- tries carry over into their national news coverage. The concept of a carbon tax is uncharacter- istic for the UK, hence it is apparently more of a discussion item in Canada and Australia. PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 15 / 27

UK: low carbon AU: uranium CA/AU: carbon tax Apart from these climate change-related features, the majority of most informative features were seemingly unrelated to climate change. Yet, they were used by the model to distinguish countries. Only a handful of features were analyzed (UK: 7, CA: 6, AU: 11), while 200 most informative features were extracted (100 positive and 100 negative) per country. Percentually, that means only 3% to 6% of features were found directly relevant to climate change. Most of these non-climate change features can be described as ortographical differences (e.g. program vs. programme), lexical variations (e.g. transport vs. transportation), local currencies (e.g. pounds vs. dollar), political terminology (e.g. treasurer vs. chancellor), national sports (e.g. foot- ball vs. soccer), and geographical and demographical terminology (e.g. province vs. state). No feasible explanation could be found for plenty of other features. The full list of extracted fea- tures is accessible as instructed under Reproducibility. In sum, our results, in line with prior research in the field, point to important differences among the countries with regard to the effects, causes, and solutions to climate change. For example, while for Australia “mining” is shown to be a distinctive cause of climate change, in the case of UK this is “flights” and for Canada “oil sands” comes up. These features seem to respond to specific economic and political contexts [34] of the countries: London serves as the

largest aviation hub in the world by passenger traffic (en.wikipedia.org/wiki/List_of_busiest_ airports_in_the_United_Kingdom), Australia is the world’s largest producer of iron ore and bauxite and the second largest of gold (en.wikipedia.org/wiki/Mining_in_Australia), and Alberta’s (Canada) oils sands, is the world’s third-largest proven oil reserve at 170 billion bar- rels (nationalgeographic.com/environment/article/alberta-canadas-tar-sands-is-growing-but- indigenous-people-fight-back). Political orientations For the political orientations model, the average F1-score was 0.80. This was somewhat lower than for the countries but still meant the vast majority of articles could correctly be classified. Hence, analysis of the most informative features was also sensible for this model. It showed the same picture as for the countries: the model relies on many features that indi- cate differences irrelevant to climate change. Moreover, the model also relies on features that were earlier found to be indicative of individual countries. This is surprising, for every country is represented on both sides of the political spectrum. However, some interesting observations can be made both literally and figuratively between the lines of these less relevant features. These observations can help answer the second research question (“What major topical and termino- logical differences can be found between political orientations and how could they be explained?”). When it comes to climate change-related features, the right and left show several contrasts.

In general, it turns out easier to find related terminology that is indicative of left-oriented newspapers. This could suggest that these also cover the subject more frequently. It also raises the question of what the model picked up instead. The share of features that seem to be country indicators or political themes is especially larger for right-oriented articles. Apart from those phrases, the features seem highly irrelevant. The relevant keywords can be characterized using a similar structure as for the model for countries: • Terminology The right and the left side of politics seem to use different words for the matter of climate PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 16 / 27

change. The right just seems to name it by the climate change. The left is more outspoken, by referring to a crisis. Furthermore, they use greenhouse and environment more. Seeing that features that were indicative of countries came back in this model, it could however be that these insights were also subject to this effect. L: greenhouse, environment, crisis R: the climate change • Causes The model also picked up on the causes of climate change. The left refers to polluters, who might in their perspective have to pay. This relates to the phrases transport and fossil fuel. The right side only seems to consider aircraft a worthy cause of the changing climate. L: polluters, transport, fossil fuel R: aircraft • Measures When it comes to measures against climate change, the right is again overtaken by the left. Only carbon tax was found for right-oriented articles. This is rather surprising, but could well mean that the right articles argue against it. Liberal journalism speaks about energy effi- ciency, climate talks and cooperation, like during COPs. Lastly, sustainable initiatives also seem to be on their agenda. L: energy efficiency, climate talks, cooperation, sustainable R: carbon tax • Effects The model did not value many features related to the effects of climate change. Only forests as found indicative for left-oriented newspapers. L: forests Again, apart from the features analyzed (L: 11, R: 3), the majority of the 100 most informa- tive features extracted for each orientation were unrelated to climate change. The share of rele-

vant features is 3% for right-oriented newspapers and 11% for left-oriented ones. Several non- climate related features that were earlier observed as informative for countries, were also infor- mative for the political orientations. Examples are local currencies, lexical variations and politi- cal differences. Moreover, specific political themes come up, that are typically favored by specific sides of the political spectrum (e.g. L: housing, schools; R: business, market). Please refer to Reproducibility, for access to the full list of extracted features. In a nutshell, our findings reveal a clear unbalance between the features of left-oriented newspapers (11) and those of the right-oriented newspapers (4). Furthermore, our results point to different causes (e.g. L: fossil fuel; R: aircraft) and measures (e.g. L: energy efficiency; R: carbon tax) between the newspapers. Thus, in line with findings by Chinn et al. [11] and Pot- thast et al. [10], our research also reveals differences due to political orientations. COP meetings The model for COP meetings scored a much lower macro average F1-score of 0.43. This inevi- tably means that the model has learned features that were not effective in distinguishing the labels in the Test data. The average F1-score comes about from a variety of F1-scores per label. Some meetings could hardly be detected (COP10, 0.18), while others still scored a fair performance (COP25, 0.62). It was considered that all labels scoring below 0.50 on F1 were too unreliable to be ana-

lyzed. Only 8 labels scored above this threshold, being COP4, 15, and 20 until 25. This list was PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 17 / 27

augmented by COP3 (F1 0.46), since the crucial Kyoto Protocol (unfccc.int/kyoto_protocol) was signed during this edition. It is remarkable that when compared to the models for countries and political orientations, a greater share of the most informative features yielded by the COP model seems to be related to climate change. This contributes to answering the research question for this aspect (“What major topical and terminological differences can be found between COP meetings and how could they be explained?”). Some COP meetings stand out with many features related to climate change, while others fall short and show many other topics. It can well be that this is caused by the varying relevance of the meetings. Some meetings seemed to have little effect, while others were quite impactful, like COP15 due to Climategate (factcheck.org/2009/12/climategate/) and COP21 for the Paris Agreement (unfccc.int/process-and-meetings/the-paris-agreement/the- paris-agreement). There seem to be some clear differences between the early and more recent COP editions. Among other things, this seems to concentrate around the terminology used to refer to climate change—especially when it comes to urgency –, the attempts made to reduce the effect, and the output of the conferences itself. Examples are provided in terms of positive (in greenþ) and negative features (in red ) for the COP meetings that featured relevant phrases. Positive features are selected by the model as characteristic for that class, while the negative ones have been identified as uncharacteristic.

COP meetings are referred to by numbers in these examples. • Terminology When it comes to the terminology used to describe climate change, a threefold shift seems to have occurred. During COP3 in 1997, the term climate change was not indicative yet. This suggests that there were concerns, though no trend of change was being described yet. Instead, merely the emissions of carbon dioxide as well as the resulting greenhouse effect and global warming seem to have been the subjects of discussion, judging by the most informative features. This changed later, seeing that the term climate change was positively informative for COP15 in 2009. The terms global warming and greenhouse were no longer highly indicative for sub- sequent COP meetings 20 and 22 respectively. For COP25 (2019), yet a new level of terminology was observed. Here, the term climate change was no longer highly indicative. To the contrary, the terms climate crisis and climate emergency were. Similarly, global warming was no longer observed in the most informative features, but the more intense global heating was. It seems the urgency of the matter started to sink in at this point. 03: greenhouse+, global warming+, carbon dioxide+, emissions at levels+, climate change {, climate { 04: climate change {, carbon {, the crisis+ 15: emissions+, climate change+, environment { 20: global warming {, the climate {, crisis { 21: climate+, crisis {, climate crisis {, climate emergency {, planet+ 22: greenhouse { 23: carbon {, green {, warming { 25: climate crisis+, climate emergency+, climate change {,

tipping points+, global heating+, global warming {, greenhouse { • Conferences Related to conferences, several features stand out as well. During the earlier meetings COP3 PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 18 / 27

and 15, summit was widely used and so was treaty. Later on, these words became negative indicators for COP22 and 23 respectively and were thus apparently not used widely any- more. It can well be that the term treaty had been replaced by climate deal and climate agree- ment at the time of COP21 and 22, since the output of COP21 was not a treaty but rather the Paris Agreement. COP21 also shows goal as informative, which likely refers to the target to limit global temperature rise to 2˚C. The meeting of COP15 was a special one, due to the occurrence of Climategate just ahead of its start. More than 1,000 stolen emails were leaked in an attempt to argue that scientists were manipulating their studies to falsely prove global warming (factcheck.org/2009/12/ climategate/). The media clearly picked up on this, judging by the occurrence of leaked, scep- tics and deniers in the most informative features. During the same edition, the concept of emissions trading became indicative. This related to the Emissions Trading System of the EU (ec.europa.eu/clima/policies/ets_en) that strives to lower European carbon emissions by the principle of cap and trade. A comparable concept is that of carryover credits. These would entitle countries to excess car- bon emissions since they previously reduced emissions further than the Kyoto Protocol required. This is especially a topic in Australia, which attempts to keep these credits valid under the new regulations of the Paris Agreement (ec.europa.eu/clima/policies/ets/credits_en). 03: summit+, conference+, treaty+, protocol {

15: emissions trading+, summit+, leaked+, cap and trade+, sceptics+, deniers+, the conference+, accord+, global agreement {, credits { 21: goal+, climate deal+, protocol {, accord {, debate { 22: summit {, conference {, policies+, emissions trading {, climate agreement+ 23: treaty { 24: emissions trading { 25: trade deal+, carryover credits+ • Effects Various effects of climate change were also found indicative of COP meetings. No real trend could be seen, apart from what might be an overall terminological change from fires towards bushfires. What is striking, is that COP25 mentions by far the most effects. This is in line with the earlier suggestion that urgency got more traction during this edition. Several terms can, to a certain extent, also be explained by natural events occurring during a given COP edition. For example, there were particularly severe bushfires in Australia (disasterphilanthropy.org/disasters/2019-australian-wildfires/) and a volcanic eruption on White Island in New Zealand (britannica.com/event/White-Island-volcanic-eruption-of- 2019) during COP25 in 2019. 03: fires+ 04: hurricane+, storm+, winds { 15: pollution {, storms {, geothermal+ 20: storm+ 21: forests {, bushfires { 22: wildlife+, quake+ 23: bushfires+ 25: bushfires+, fires+, eruption+, smoke+, volcano+ PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 19 / 27

• Causes Where there are effects, there are causes. Logically, several of these were found within the features for the COP meetings. Again, it was hard to notice a blatant trend in these. The explosion for COP25 was not observed either. Rather, more features were found for interme- diate meetings like COP20 and 21. It may be that causes were not specifically addressed before and are more left to common sense by now. In extension, the attention may increas- ingly be drawn towards measures against climate change. 03: diesel {, coal+, petrol { 04: coal { 15: pipeline { 20: shale oil+, methane+, gas {, industrial { 21: fracking+, fuels+, transport {, fossil+ 22: pipeline+, coal+ 23: diesels+, plastic+ 24: gas { 25: car { • Measures Looking at measures, none were found for earlier COP meetings. This could support the idea that the attention is shifting from observing causes to acting with measures. COP21 clearly shows solar and clean energy as indicative. COP23 features energy storage. By COP25, clean energy had become a negative feature, thus being uncharacteristic for the class. Instead, transition, electric and emissions reduction had become positively indicative, i.e. characteris- tic for articles published around COP25. 20: reduce { 21: solar+, clean energy+ 23: energy storage+, uranium { 24: power stations { 25: transition+, electric+, clean energy {, emissions reduction+ Only 3% to 19% of the extracted most informative features were deemed relevant to climate change and have thus been analyzed (3: 14, 4: 7, 15: 17, 20: 14, 21: 18, 22: 10, 23: 14, 24: 3, 25:

19). While most features cannot be plausibly explained, several can be related to historical events co-occuring with COP meetings. Examples are the Ottawa Treaty (un.org/ disarmament/anti-personnel-landmines-convention/) on land mines (COP3), the Ebola pan- demic in Western Africa (COP20), the advance of Islamic State in Iraq and Syria (COP21), the presidential election of Donald Trump (COP22), the mass displacement of the Rohingya (COP23), the Brexit agreement (COP24), and the impeachment of Trump (COP25). All fea- tures can be accessed, as indicated under Reproducibility. Apart from a classification model, the COP meetings were also predicted using regression. While the classification model had yielded rather low scores, the regression model seemed to be performing quite well. This results from the fundamentally different approach of the two algorithms and the related means of evaluation. Whereas a prediction of a class can only be wrong or right, a numeric prediction by regression is evaluated with its precise error from the gold standard. Interpreting the MAE of 2.95, one can say that the regression model is on average roughly 3 COP editions off. RMSE corrects for larger variance and is 4.07 for this model. This can PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 20 / 27

roughly be explained as the model being 4 editions off, although the metric is in fact more complex. Considering that there are 26 COP meetings, not every edition is as impactful and most characteristics change gradually rather than at once, these scores shall be considered acceptable. The most informative features of regression models are only based on a single set of positive and negative coefficients. Since n-grams are represented by positive tf-idf importance scores, these respectively refer to an increase or a decrease of the predicted COP meeting number. Looking at the top of the positive and negative features, conclusions can therefore only be drawn in terms of phrases for later and earlier COP meetings. This seems to reflect many of the findings done with the classification model. Several examples are given below, for earlier (negative features, in red ) and later COPs (positive features, in greenþ). • Terminology The terminology found using classification comes back in the regression model. Terminol- ogy has shifted to become more descriptive of a trend (greenhouse vs. climate) and more extreme (climate crisis). It also seems that carbon dioxide is now more frequently referred to merely as carbon. Earlier: greenhouse, global warming, greenhouse emissions, carbon dioxide, environmentalists, earth, climatic, environment, greenhouse effect, carbon dioxide emissions Later: climate, planet, carbon, climate crisis, emissions, CO2, footprint, climate emergency, carbon emissions • Causes

Similarly, phrases that describe the causes of climate change largely matched those found before. It does seem that describing causes is something of the later editions. However, this is in fact sound with the findings that intermediate meetings like COP20 until 23 are character- ized more in terms of causes. Earlier: petroleum, gas, traffic Later: aircraft, shale, fracking, diesel, air pollution, plastic, coal, methane, drilling, vehicles, shale oil, oilsands • Effects Some effects were found in the most informative features, though much less than for the classification model. An interesting word that is yielded, is ozone, referring to the ozone layer hole. This was more of a concern earlier but seems to be on its return under the effect of the Montreal Protocol (unep.org/ozonaction/who-we-are/about-montreal-protocol). It is shown that the media wrote a lot more about it during earlier COP meetings. Earlier: ozone, smog, air conditioning Later: bushfires • Measures The potential trend that measures are becoming more and more subject of attention seems to be reflected by the yields of the regression model. No features that describe measures were found on the negative side of the coefficient spectrum, whereas quite some could be detected with positive scores. These phrases are predominantly the same as for classification. Though sustainability and electric vehicles are interesting additions. Later: electric, electric vehicles, battery, and solar, renewables, sustainability, emissions reduction PLOS ONE Learning from climate change news

PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 21 / 27

Like the classification models, the regression model mostly produced features unrelated to climate change. These are much like those yielded in classification, but were not analyzed in- depth. Of the 250 positive and negative features extracted, 6 to 12% were analyzed as relevant to climate change (Earlier: 16, Later: 29). The full list of features is available, as described under Reproducibility. Discussion & conclusions This study has uncovered relevant topical and terminological differences between newspaper articles on climate change in Australia, Canada, and the United Kingdom. In doing so, it has made use of supervised machine learning. Since most of the studies on climate change news focus on single countries and make use of manual content analysis [17], we believe that our research can positively contribute to the development of the field. The study aimed to find a response to three research questions: Are there major topical and terminological differences between the articles on climate change by newspapers from differ- ent countries (RQ1)?; Are there major topical and terminological differences between the arti- cles on climate change by newspapers with different political orientations (RQ2)?; Are there major topical and terminological differences between the newspaper articles on climate change during different COP meetings (RQ3)? The results of our machine learning model reveal that countries can be automatically classi- fied accurately, with a macro F1-score of 0.90. Based on these findings and the analysis of most

informative features, it can also be concluded that there are topical and terminological differ- ences between these countries (RQ1). These differences have to do with climate change termi- nology, and its effects, causes, and measures. The model for political orientations had the second best fit, scoring a macro F1-score of 0.80. This result allows us to conclude that it is possible to automatically classify articles into the political orientation of their newspaper. This means that distinctive features (topical and terminological) for political orientation also exist in the corpus (RQ2). These differences, in line with the findings reached in the country model, are terminological and have to do with the causes, measures, and effects of climate change news articles. The results of the models for differentiating between COP meetings are slightly less outspo- ken. It is not possible to convincingly distinguish every COP edition. Indeed, the macro F1 score was 0.43, although it greatly differed between the meetings. This signals that not every meeting was equally relevant. However, it can be concluded that the model detected certain differences, seeing that the baseline was outperformed. When assessed as a regression task, taking into account the temporal nature of these labels, an RMSE of 4.07 was yielded. Thus, the COP meeting can properly be estimated up to roughly half a decade. Regression shows to be a more suitable method to predict the COP meeting for newspaper articles. This leads us to conclude that it is possible to automatically predict the

COP meeting sufficiently accurately. Some COP meetings do indeed show relevant topical and terminological differences, while others hardly do (RQ3). Those that do feature these differences show a clear terminology tran- sition over the years and differences in conference-related phrases, as well as effects, causes, and measures. It also becomes clear that later COPs show more relevant features and more urgent phrases than earlier ones. Specifically, terminology has shifted to become more descrip- tive of a trend (greenhouse vs. climate) and more extreme (climate crisis); describing causes is something of the later editions (e.g., fracking, drilling, oilsands); and measures are becoming more and more subject of attention (e.g., electric vehicles, renewables, emissions reductions). PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 22 / 27

The latter is in line with findings by Keller et al. that climate change news is individualizing, with a growing focus on individual actions [14]. Apart from features, this study has yielded a new dataset that can be analyzed in many ways to uncover new insights concerning newspapers’ approaches to climate change. The data also lends itself for e.g. headline generation or manual qualitative analyses. Future research could for example make use of neural networks [45] to outperform our current models or employ word embeddings [44] to predict the political orientation of the newspapers. On the other hand, studying the data through the use of manual discourse analysis will help unfold different insights that automated methods struggle to detect, e.g. some contextual characteristics of the corpus. All in all, the research constitutes a first step in exploring the automatic detection of differ- ences in news articles with regard to countries, political orientations, and COP meetings. The study contributes to the use of supervised machine learning for highlighting discriminative features on how newspaper articles approach climate change globally. Future works should pick up where this study left off. Above all, we consider it essential to further bridge our results to a wide array of previous works, and see plenty of opportunities to do so. For example, studies focusing on scientific publications [6, 38, 41], government policies [7, 39, 42] or news articles [11, 14–17, 26, 30, 31, 35]. It will be insightful to put the features

yielded in the study at hand further into context and to see whether and how results align with earlier findings. Furthermore, future works should expand our analyses to other countries and news articles from the most recent COP meetings (26, 27 and 28). Now that the feasibility of the approach has been proven, we also consider it worthwhile to focus on further finetuning machine learn- ing model development in ways we did not cover, e.g. by tuning the SVM’s hyperparameters (scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) and using cross-validation (scikit-learn.org/stable/modules/cross_validation.html). The creation of a protocol for feature analysis should also be taken into account, to further structure annotation of the yielded most informative features. Another opportunity lies in more directly comparing feature overlap from our classification and regression models for COP meetings. Limitations While our exploratory research has yielded interesting insights into how news media are approaching climate change across countries, we also acknowledge several limitations that accompany our investigation. First, we consider machine learning a way to reduce the chance of introducing human bias in the analysis of newspaper articles and have presented the technique as such. Our methodol- ogy proves that it is indeed possible to systematically find, collect, and digest articles using clas- sification and regression models. Nonetheless, the analysis of these features was a manual process, based on our own general knowledge and common sense. We are convinced that

more systematic approaches to analyzing the meaning and context of features yielded by our models can be designed. For example, we strongly suggest bridging our results with findings by preceding publications on climate change news [11, 14–17, 26, 30, 31, 35], scientific publi- cations [6, 38, 41] or government policies [7, 39, 42]. Also, we stress the need to pair the fea- tures we present with the knowledge that experts in the field of climate change have. It must also be pointed out that the machine learning methodology presented in this study does not take into account context beyond the features themselves. Word order is only pre- served within multigram features. Beyond that, it is impossible to determine from the model itself how certain most informative features are used in the original text and substantiate why PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 23 / 27

they appear as significant for a given class. It may be that a feature that is significant to the model is not so apparent to a human reader, e.g., due to its context or frequency. We believe this can also be overcome by further analysis, be it using a deeper analysis of the dataset we cre- ated, or using a broader comparison with other findings. We see analysis of most informative features yielded by machine learning models as a useful addition to the field of climate change. We would like to encourage researchers in this field to incorporate the techniques we presented into their own qualitative studies. This could improve reliability and uncover patterns that may otherwise remain unseen by the human eye. We do not mean to suggest that manual analysis can be entirely replaced by machine learning. Rather, we believe the two can complement each other. A bottom-up approach such as the one we have presented, in which the content is leading, can prompt researchers to start looking for dif- ferences they might overlook in top-down analyses. Second, a more technical limitation of this proof-of-concept approach is the lack of parame- ter tuning for both the classification and regression models, e.g. using an RBF instead of a lin- ear kernel, or optimizing C for the support vector machines classifier. We have focused on collecting the data and optimizing the preprocessing of features, but have deliberately chosen not to involve the parameters of the employed LinearSVC and LinearRegression esti-

mators. It may well be that parameter tuning can further enhance the performance of the mod- els and can thus lead to more representative features. We therefore highly recommend incorporating even the mildest parameter tuning into future work. We also believe that the preprocessing strategy designed in this study can further be improved upon. For example, features that differ e.g. orthographically, lexically and geographi- cally, and are currently used by the model to distinguish countries should be filtered out more effectively. Named entity recognition also leaves room for improvement, which can possibly be achieved by using more recent, larger (transformer-based), and news-related models from SpaCy. Lastly, our systematic data collection approach was based on 3 co-occurring keywords extracted from Wikipedia articles, to raise relevancy towards climate change. It can however not be guaranteed that all collected articles explicitly cover the topic. Ambiguity of keywords and tricky keyword combinations may have captured articles on other subjects as well. We deliberately chose not to further reduce the size of the dataset by filtering for 4 co-occurring keywords, aiming to balance accuracy and data size. It was considered that a larger amount of data could mostly cancel out noise from less relevant articles. Despite the limitations that this study faced, we do consider it a successful proof of concept for feature analysis based on machine learning models in the field of climate change. We hope

that future work will overcome several of these limitations, but moreover that machine learn- ing will start to become a useful tool for researchers in analyzing textual patterns in climate change news and related documents. Supporting information S1 Appendix. Country selection. Detailed explanation of the country selection process. (PDF) S2 Appendix. Keyword selection. Figures detailing the keyword selection process. (PDF) S3 Appendix. Data description. Table of further statistics describing the data. (PDF) PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 24 / 27

S4 Appendix. Feature selection. Further details on the feature selection process. (PDF) S5 Appendix. Experiments & performance. Tables detailing model experiments and the resulting performance. (PDF) Acknowledgments This study was presented at the Environmental Communication division of the 73rd Annual ICA (icahdq.org) Conference (ICA23). We are thankful for all the useful comments we received on this presentation. We thank the Center for Information Technology of the Univer- sity of Groningen for providing access to the Peregrine and Hábrók high-performance comput- ing clusters (rug.nl/society-business/centre-for-information-technology/research/services/ hpc/facilities/peregrine-hpc-cluster). Author Contributions Conceptualization: Stijn Eikelboom, Marc Esteve-Del-Valle, Malvina Nissim. Data curation: Stijn Eikelboom. Formal analysis: Stijn Eikelboom. Investigation: Stijn Eikelboom. Methodology: Stijn Eikelboom, Malvina Nissim. Project administration: Stijn Eikelboom, Marc Esteve-Del-Valle, Malvina Nissim. Resources: Stijn Eikelboom. Software: Stijn Eikelboom. Supervision: Marc Esteve-Del-Valle, Malvina Nissim. Validation: Stijn Eikelboom, Marc Esteve-Del-Valle, Malvina Nissim. Visualization: Stijn Eikelboom. Writing – original draft: Stijn Eikelboom. Writing – review & editing: Stijn Eikelboom, Marc Esteve-Del-Valle. References 1. Boykoff M, Luedecke G. Elite news coverage of climate change. In: Oxford Research Encyclopedia of Climate Science. Oxford University Press; 2016.

2. Moser SC. Communicating climate change: history, challenges, process and future directions. Wiley Interdisciplinary Reviews: Climate Change. 2010; 1(1):31–53. 3. Carmichael JT, Brulle RJ. Elite cues, media coverage, and public concern: an integrated path analysis of public opinion on climate change, 2001–2013. Environmental Politics. 2017; 26(2):232–252. https:// doi.org/10.1080/09644016.2016.1263433 4. Weingart P, Guenther L. Science communication and the issue of trust. Journal of Science communica- tion. 2016; 15(5):C01. https://doi.org/10.22323/2.15050301 5. Carvalho A. Media (ted) discourses and climate change: a focus on political subjectivity and (dis) engagement. Wiley Interdisciplinary Reviews: Climate Change. 2010; 1(2):172–179. 6. Sietsma AJ, Ford JD, Callaghan MW, Minx JC. Progress in climate change adaptation research. Envi- ronmental Research Letters. 2021; 16(5):054038. https://doi.org/10.1088/1748-9326/abf7f3 PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 25 / 27

7. Sietsma AJ, Callaghan M, Biesbroek R, Theokritoff E, Thomas A, Canosa IV, et al. Global Tracking of Climate Change Adaptation Policy Using Machine Learning: a Systematic Map Protocol. Protocol Exchange. 2022;. 8. Arlt D, Hoppe I, Wolling J. Climate change and media usage: Effects on problem awareness and beha- vioural intentions. International Communication Gazette. 2011; 73(1-2):45–63. https://doi.org/10.1177/ 1748048510386741 9. Nisbet EG, Manning M, Dlugokencky E, Fisher R, Lowry D, Michel S, et al. Very strong atmospheric methane growth in the 4 years 2014–2017: Implications for the Paris Agreement. Global Biogeochemi- cal Cycles. 2019; 33(3):318–342. https://doi.org/10.1029/2018GB006009 10. Potthast M, Kiesel J, Reinartz K, Bevendorff J, Stein B. A stylometric inquiry into hyperpartisan and fake news. arXiv preprint arXiv:170205638. 2017;. 11. Chinn S, Hart PS, Soroka S. Politicization and polarization in climate change news content, 1985-2017. Science Communication. 2020; 42(1):112–129. https://doi.org/10.1177/1075547019900290 12. Basile A, Dwyer G, Medvedeva M, Rawee J, Haagsma H, Nissim M. N-gram: New Groningen author- profiling model. arXiv preprint arXiv:170703764. 2017;. 13. van Dalen RG, Melein LR, Plank B. Profiling Dutch authors on Twitter: Discovering political preference and income level. Computational Linguistics in the Netherlands Journal. 2017; 7:79–92. 14. Keller TR, Hase V, Thaker J, Mahl D, Scha¨ fer MS. News Media Coverage of Climate Change in India

1997–2016: Using Automated Content Analysis to Assess Themes and Topics. Environmental Commu- nication. 2019; p. 1–17. 15. Grundmann R, Scott M. Disputed climate science in the media: Do countries matter? Public Under- standing of Science. 2014; 23(2):220–235. https://doi.org/10.1177/0963662512467732 PMID: 23825269 16. Holt D, Barkemeyer R. Media coverage of sustainable development issues–attention cycles or punctu- ated equilibrium? Sustainable development. 2012; 20(1):1–17. https://doi.org/10.1002/sd.460 17. Schmidt A, Ivanova A, Scha¨ fer MS. Media attention for climate change around the world: A comparative analysis of newspaper coverage in 27 countries. Global Environmental Change. 2013; 23(5):1233– 1248. https://doi.org/10.1016/j.gloenvcha.2013.07.020 18. Boyce T, Lewis J. Climate change and the media. Peter Lang; 2009. 19. Boykoff J. US media coverage of the Cancun climate change conference. PS: Political Science & Poli- tics. 2012; 45(2):251–258. 20. Butler C, Pidgeon NF. Media communications and public understanding of climate change: reporting scientific consensus on anthropogenic warming. Global Crises and the Media. 2009; 5. 21. Davidsen C, Graham D. Newspaper reporting on climate change, green energy and carbon reduction strategies across Canada 1999–2009. American Review of Canadian Studies. 2014; 44(2):151–168. https://doi.org/10.1080/02722011.2014.914047 22. Dirikx A, Gelders D. Global warming through the same lens: an explorative framing study in Dutch and French newspapers. Climate change and the media. 2009; p. 200–210.

23. Doulton H, Brown K. Ten years to prevent catastrophe?: Discourses of climate change and international development in the UK press. Global Environmental Change. 2009; 19(2):191–202. https://doi.org/10. 1016/j.gloenvcha.2008.10.004 24. Lockwood A. Preparations for a post-Kyoto media coverage of UK climate policy. Climate change and the media. 2009; p. 186–199. 25. Olausson U. Global warming—global responsibility? Media frames of collective action and scientific cer- tainty. Public understanding of science. 2009; 18(4):421–436. https://doi.org/10.1177/ 0963662507081242 26. Hase V, Mahl D, Scha¨fer MS, Keller TR. Climate change in news media across the globe: An automated analysis of issue attention and themes in climate change coverage in 10 countries (2006–2018). Global Environmental Change. 2021; 70:102353. https://doi.org/10.1016/j.gloenvcha.2021.102353 27. Painter J, Scha¨fer MS. Global similarities and persistent differences: a survey of comparative studies on climate change communication. Climate change and the media. 2018; p. 39–58. 28. Hart PS, Feldman L. The impact of climate change–related imagery and text on public opinion and behavior change. Science Communication. 2016; 38(4):415–441. https://doi.org/10.1177/ 1075547016655357 29. Kunelius R, Eide E, Tegelberg M, Yagodin D, editors. Media and global climate knowledge: Journalism and the IPCC. United Kingdom: Palgrave Macmillan; 2017. 30. Barkemeyer R, Figge F, Hoepner A, Holt D, Kraak JM, Yu PS. Media coverage of climate change: An

international comparison. Environment and Planning C: Politics and Space. 2017; 35(6):1029–1054. PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 26 / 27

31. Vu HT, Liu Y, Tran DV. Nationalizing a global phenomenon: A study of how the press in 45 countries and territories portrays climate change. Global Environmental Change. 2019; 58:101942. https://doi. org/10.1016/j.gloenvcha.2019.101942 PMID: 32288343 32. Wozniak A, Wessler H, Chan CH, Lueck J. The event-centered nature of global public spheres: The UN climate change conferences, Fridays for future, and the (limited) transnationalization of media debates. International Journal of Communication. 2021; 15:27. 33. Wessler H, Wozniak A, Hofer L, Lu¨ck J. Global multimodal news frames on climate change: A compari- son of five democracies around the world. The International Journal of Press/Politics. 2016; 21(4):423– 445. https://doi.org/10.1177/1940161216661848 34. Reese SD. Understanding the global journalist: A hierarchy-of-influences approach. Journalism studies. 2001; 2(2):173–187. https://doi.org/10.1080/14616700118394 35. Stecula DA, Merkley E. Framing climate change: economics, ideology, and uncertainty in American news media content from 1988 to 2014. Frontiers in Communication. 2019; 4:6. https://doi.org/10.3389/ fcomm.2019.00006 36. Fischer AP, Callaghan M, Berrang-Ford L, Nielsen M, Sotnik G, Canosa IV, et al. The global adaptation mapping initiative (GAMI): Part 2–screening protocol. Protocol Exchange. 2021;. 37. Webersinke N, Kraus M, Bingler JA, Leippold M. Climatebert: A pretrained language model for climate- related text. arXiv preprint arXiv:211012010. 2021;.

38. Berrang-Ford L, Sietsma AJ, Callaghan M, Minx JC, Scheelbeek PF, Haddaway NR, et al. Systematic mapping of global research on climate and health: a machine learning review. The Lancet Planetary Health. 2021; 5(8):e514–e525. https://doi.org/10.1016/S2542-5196(21)00179-0 PMID: 34270917 39. Biesbroek R, Badloe S, Athanasiadis IN. Machine learning for research on climate change adaptation policy integration: An exploratory UK case study. Regional Environmental Change. 2020; 20(3):85. https://doi.org/10.1007/s10113-020-01677-8 40. Slapin JB, Proksch SO. A scaling model for estimating time-series party positions from texts. American Journal of Political Science. 2008; 52(3):705–722. https://doi.org/10.1111/j.1540-5907.2008.00338.x 41. Berrang-Ford L, Siders A, Lesnikowski A, Fischer AP, Callaghan MW, Haddaway NR, et al. A system- atic global stocktake of evidence on human adaptation to climate change. Nature Climate Change. 2021; 11(11):989–1000. https://doi.org/10.1038/s41558-021-01170-y 42. Sietsma AJ, Theokritoff E, Biesbroek R, Villaverde Canosa I, Thomas A, Callaghan M, et al. Climate Change Adaptation Policy Across Scales: A Machine Learning Evidence Map. Available at SSRN 4318820. 2023;. 43. Manning C, Schutze H. Foundations of statistical natural language processing. MIT press; 1999. 44. Almeida F, Xexe´ o G. Word embeddings: A survey. arXiv preprint arXiv:190109069. 2019;. 45. Sarle WS. Neural networks and statistical models. Proceedings of the 19th Annual SAS Users Group International Conference. 1994; 13. PLOS ONE Learning from climate change news

PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 27 / 27