Learning from climate change news: Is the world on the same page?

RESEARCH ARTICLE Learning from climate change news: Is the world on the same page? Stijn EikelboomID1☯¤*, Marc Esteve-Del-ValleID2☯¤, Malvina Nissim3☯¤ 1 Department of Language Technology, Faculty of Arts, University of Groningen, Groningen, The Netherlands, 2 Research Centre for Media and Journalism Studies, Faculty of Arts, University of Groningen, Groningen, The Netherlands, 3 Center for Language and Cognition, Faculty of Arts, University of Groningen, Groningen, The Netherlands ☯ These authors contributed equally to this work. ¤ Current address: Faculty of Arts, University of Groningen, Groningen, The Netherlands * s.e.eikelboom@rug.nl Abstract Climate change challenges countries around the world, and news media are key to the pub- lic’s awareness and perception of it. But how are news media approaching climate change across countries? With the problem of climate change and its solution being global, it is key to determine whether differences in climate change news reports exist and what they are across countries. This study employs supervised machine learning to uncover topical and terminological differences between newspaper articles on climate change. An original data- set of climate change articles is presented, originating from 7 newspapers and 3 countries across the world, and published in English during 26 Conference of the Parties (COP) meet- ings from the United Nations Framework Convention on Climate Change (UNFCC). Three aspects are used to discriminate between articles, being (1) countries, (2) political orienta-

tions, and (3) COP meetings. Our results reveal differences with regard to how newspaper articles approach climate change globally. Specifically, climate change-related terminology of left-oriented newspapers is more prevalent compared to their right-oriented counterparts. Also, over the years, newspapers’ climate change-related terminology has evolved to con- vey a greater sense of urgency. Introduction Of the many challenges that mankind will face during the coming decades, it is likely that cli- mate change will be one of the most pressing and demanding. Since the late 19th century, aver- age global temperature has already risen by 1.1 degrees Celsius, which can largely be attributed to human-made emissions (climate.nasa.gov/evidence/). The sixth assessment report (AR6) of the Intergovernmental Panel on Climate Change (IPCC) points out that there is a more than 50% chance that global temperature rise will reach or surpass 1.5 degrees Celsius between 2021 and 2040 (ipcc.ch/report/sixth-assessment-report-cycle/). Therefore, climate change shall be high on the global agenda of scientific research, politics, and eventually every citizen. An important means of calling attention to climate change is the (usually) yearly gathering of the Conference of the Parties (COP) from the United Nations Framework Convention on PLOS ONE PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 1 / 27 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Eikelboom S, Esteve-Del-Valle M, Nissim M (2024) Learning from climate change news: Is

the world on the same page? PLoS ONE 19(3): e0297644. https://doi.org/10.1371/journal. pone.0297644 Editor: Michael Flor, Educational Testing Service (ETS), UNITED STATES Received: March 27, 2023 Accepted: January 9, 2024 Published: March 20, 2024 Copyright: © 2024 Eikelboom et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: The data used in this study was collected from Nexis Uni under license of the University of Groningen (UG). We are not the owner of the data. As we have verified with the Digital Competence Center of the UG, it can therefore not be shared publicly for legal reasons, however much we would like to. All code employed, plots generated and most informative features yielded in this study are available on DataverseNL: https://doi.org/10.34894/AMISPR. Together with the Methodology (e.g., steps and queries) in the manuscript, this is everything required by licensees to reproduce the data.

Climate Change (UNFCCC) (unfccc.int/process/bodies/supreme-bodies/conference-of-the- parties-cop). The convention strives to prevent “dangerous” human interference with the cli- mate system (unfccc.int/process-and-meetings/the-convention/what-is-the-united-nations- framework-convention-on-climate-change) and the meetings are the chief platform for key decision-making. These have occurred 26 times between 1995 and 2019. This study approaches climate change from a linguistic perspective. Language plays an important role in tackling the numerous effects of climate change. It facilitates precarious problem-solving during global political summits, it is used in scientific papers that must con- vince politicians to act, and it spreads the news to the general public [1–7]. Precisely the last context is considered crucial in this study. After all, behavioral change by every citizen is key to reducing greenhouse gas emissions. Politics and businesses must play a key role in facilitating this, but for citizens to really change their comings and goings, intrinsic conviction and motivation are essential. News media are an important factor here, as they are the major source of climate informa- tion for most citizens [1], raising their understanding and shaping their perception [2], as well as impacting behaviour [8]. Politically, news media set the policy agenda [3, 4] and influence public debate [5, 9]. Moreover, climate change is fundamentally a global issue and not limited to country bor- ders. It requires sustainable collaboration between countries and a consistent international

field, by revealing differences that might not be picked up by experts in manual analysis. Specifically, the goal of this study is to detect topical and terminological differences between international newspaper articles on climate change. Since several aspects can lead to such dif- ferences, three characteristics are used to discriminate between the articles which thus become the prediction targets of the machine learning models. These are (1) the country of the newspa- per, (2) the political orientation of the newspaper and (3) the COP meeting occurring during article publication. For their annual occurrence, the COP meetings are an ideal reflection of the moment of publication. Moreover, the meetings boost the publication of climate-related articles [14]. Machine learning is used to automatically conduct the analyses. Each aspect is separately analyzed through classification. Since COP meetings have a temporal nature, this aspect is also analyzed with regression. The resulting models are intended to gain more insight into key dif- ferences between articles. Hence, human-understandable features are used within statistical PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 2 / 27 Funding: The author(s) received no specific funding for this work. Competing interests: The authors have declared that no competing interests exist.

models. These are explicitly favored over more complex features and architectures that could improve performance, but are hard to interpret, such as word embeddings and neural networks. Three research questions are answered: 1. Are there major topical and terminological differences between the articles on climate change by newspapers from different countries? 2. Are there major topical and terminological differences between the articles on climate change by newspapers with different political orientations? 3. Are there major topical and terminological differences between the newspaper articles on cli- mate change during different COP meetings? It is hypothesized that differences can be found between newspaper articles from different countries, particularly between countries from different continents [15, 16]. A larger contrast is expected to be seen between right and left-oriented newspapers, considering the issue seems to become ever more politicized [11]. Lastly, it is expected that differences exist between COP meetings as well, especially between editions that are further apart [14, 17]. Previous work Numerous studies have stressed the importance of mass media in communicating the con- cerns associated with climate change [15, 18–25]. Furthermore, news media can influence pub- lic concern about climate change by increasing levels of attention to it. Journalists, as Hase et al. contend [26], can present climate change as a more ubiquitous topic in the news by (a) stressing the sociological dimension of climate change and illustrating how humans are aware

of, affected by, combating, or causing climate change [27], (b) focusing on how climate change affects public health [9] or (c) on activities that people can take [28], thereby encouraging pub- lic participation. Climate change coverage can be distinguished by both domestic and global viewpoints on what causes it, who is affected by it, and how it can be addressed [29]. Several studies have been conducted to determine whether media coverage differs around the globe [30, 31]. According to [32], finding cross-national commonalities in the way climate change is reported is key because the existence of these commonalities may suggest the emergence of a public sphere as “an enduring structure that enables political debate and opinion formation for and with a global audience” (p. 689). However, research generally finds that discrepancies in cli- mate change coverage persist [33]. These variations could be brought on by global factors that affect news media coverage, such as the political or economic contexts of various nations [34]. For instance, different national responsibilities for addressing climate change, politics, or vul- nerabilities may have an impact on how much attention is paid to and how much is covered by the media [17]. According to an overview by Schmidt et al. [17], the majority of studies on climate change news focused on single countries, and only a few assessed differences between newspapers across nations. Also, among the studies assessing single and cross-country newspapers, manual

content analysis seems to be the preferred method. As for the automated analysis of climate change-related news media, most of the relatively slim research on the topic relied on auto- mated content analysis techniques [16, 17], while very little used machine learning [11, 14, 35]. Thus, the state of the art of the literature seems to indicate both the necessity to expand the cross-country analysis of climate change-related news media and the need to do so by explor- ing the use of machine learning techniques. PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 3 / 27

Certainly, machine learning can play a pivotal role in the analysis of vast climate change- related datasets. This is because machine learning enables the input of vast volumes of data into a computer algorithm, empowering the system to examine this data and generate data- driven recommendations and decisions solely based on the input information (see, for exam- ple, [6, 13, 36–39]). Consequently, the utilization of machine learning helps reduce human intervention in data manipulation and reduces the chance of introducing biases. Grundmann and Scott [15] performed one of the first automated content analysis studies into the discourse of climate change over time. They used newspapers from France, Germany, the UK, and the USA. A corpus of 106.5 million words was created, by retrieving articles pub- lished between 2000 and 2010 from Lexis Nexis. The peaks and troughs of attention for climate change were visualized per country. Furthermore, keyword analysis such as collocation analysis was also employed. The goal of the research was to study which terminology each country favored to describe the concept of climate change. The authors also compared the attention paid to advocates and skeptics and analyzed the influence of Climategate (factcheck.org/2009/ 12/climategate/). The study revealed clear differences between the climate change terminology among the four countries. The study also showed that advocates were favored over skeptics in all countries, but more in Germany and the UK than in France and the USA. However, this

study could methodologically only rely on manual detection of climate change-related key- words, risking introducing human biases and missing out on other potentially relevant aspects of the data. Holt and Barkemeyer [16] pioneered the broadening of the geographical scope, as they con- sidered 112 newspapers from 39 countries. This is one of the largest samples ever considered, also according to Schmidt et al. [17]. Articles were published between 1990 and 2008 and were in seven languages. For each newspaper, the monthly averages of hits on the search keywords were calculated. These allowed to quantitatively compare the coverage of sustainability and cli- mate change over time. The analysis showed clear peaks for sustainability around the confer- ences of Rio de Janeiro (1992), Kyoto (COP3, 1997), and Johannesburg (2002). For both sustainability and climate change, the trend was upwards. The latter term showed a sudden upsurge after 2007. Furthermore, the authors identified several interesting local phenomena, such as the Australian coverage being up to five times larger than the global average and the US trend being downward, thus opposing the global one. Moreover, fluctuations over time were successfully interpreted by means of the issue-attention cycles and punctuated equilibrium models. This highlighted events such as the release of Al Gore’s movie An Inconvenient Truth in 2006 and the financial crisis starting in 2008. While these are interesting insights, the size of the dataset left the authors seemingly no choice but to use a quantitative representation of cov-

erage. Certainly, they could relate peaks and troughs to events, but frequency alone can never reveal nuances within the text. Fewer publications on climate change can for example still imply more factual content, featuring less skepticism. This asks for a deeper analysis of the contents, yet without having to give up on data size. In yet another quantitative study, Schmidt et al. [17] employed newspaper articles from 27 countries, including both countries with commitments to the Kyoto Protocol and countries that suffer strongly from the consequences of climate change. Where possible, two newspapers were incorporated from each country, to reflect different political orientations. The relevant articles from these newspapers were retrieved by using a complex search string, aggregating a wide range of terms related to climate change. In doing so, the selection was more fine- grained; articles that covered climate change without literally mentioning the term could be included while preventing noise from articles that used words like temperature or heat in another context. In the study, issue attention level, defined as the attention for climate change- related topics compared to the attention for other issues, was handled as a key metric. Apart PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 4 / 27

from detecting an overall increase of climate change attention over time, Schmidt et al. [17] also found cross-national differences in the attention levels. Further analysis of these differ- ences indicated that more vulnerable countries do not necessarily deviate from the average. Additionally, countries that have obligations under the Kyoto Protocol also have more issue attention to climate change on average. This approach yields, however, similar concerns as the study by Holt and Barkemeyer [16]: word frequency is certainly interesting, yet it cannot serve as a full representation of content just on its own. When applied correctly, this is something machine learning can overcome. Keller et al. [14] were among the first to employ machine learning (ML), albeit unsuper- vised. They analyzed Indian newspaper articles for 28 topics related to climate change. The authors collected 18,224 articles from the two largest Indian newspapers, published between 1997 and 2016. In line with previous research in the field [15, 17], three keywords were used to find the articles, being climate change, global warming and greenhouse effect. They first assessed the proportion of articles dealing with climate change. Then, topic modeling with LDA was applied using unsupervised machine learning. The authors found that peaks in reporting on climate change strongly coincide with COP meetings (unfccc.int/process/bodies/supreme- bodies/conference-of-the-parties-cop). Overall, they revealed that reporting on climate change

increased substantially between 1997 and 2016. Nonetheless, the attention in India still lagged way behind compared to other countries. Furthermore, they identified 28 climate change- related topics. Agriculture was the most popular at 5.02%. These topics could be structured under four overarching themes: Climate Change and Society (27.37%) was most popular over time, followed by Climate Politics (23.12%). In recent years, both became slightly less common, while Climate Change Impacts (17.75%) became more frequent. The less popular Climate Change Science (6.24%) saw rather few fluctuations. Lastly, they observed the individualization of climate change news, showing that more and more attention is drawn to individual effects and actions. Unsupervised machine learning was also used by Chinn et al. [11]. The authors built a data- set of newspaper articles published between 1980 and 2017 in the United States, covering both national and regional newspapers. The articles were retrieved from Lexis Nexis by searching for the Environment tag and annotated using seven keyword dictionaries on different climate- related topics. The labeled dataset was then used to study the politicization of climate change news. Mentions of Republicans and Democrats were quantitatively compared to mentions of scientists. Their results show an increase for politicians, against a decrease for scientists. In fact, these two frequencies crossed between 2000 and 2010; every article since 2000 on average con- tained at least one mention of politicians, confirming the politicization trend. Additionally, the

authors employed unsupervised machine learning to study polarization. First, partisan para- graphs on climate change were identified, i.e. mentioning either Republicans or Democrats. Wordfish (tutorials.quanteda.io/machine-learning/wordfish/) [40] was then used to determine a score for the linguistic differences between the two paragraphs. This was done for all included two-year periods corresponding to the terms of US Congress, enabling comparison over time. Results showed consistent linguistic polarization until 2011, but an increase after- ward, as shown by the mentions of Republican terminology, which overtook coverage of Dem- ocratic phrases. More recently, Hase et al. [26] conducted a novel longitudinal automated cross-country analysis of news media across the globe. They compared coverage in ten countries from the Global North and the Global South by employing unsupervised machine learning. Their study revealed both similarities and differences in coverage of climate change. Political, scientific, and (partly) societal focusing events were related to small mounts in global news media inter- est and countries from both areas reported on “ecological changes, climate science and the PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 5 / 27

societal dimension of climate change” (p. 10). However, their study also points to important differences between countries of the Global North and the Global South, such as the first focus- ing more on climate science while the second paying more attention to the impact of climate change on people’s lives. Stecula and Merkley [35] are among the few researchers who have employed supervised machine learning with support vector machines (SVMs) in the field of climate change news. They collected a dataset from four US news sources, being newspapers (1) The New York Times, (2) The Wall Street Journal and (3)The Washington Post, plus the news wire agency (4) Associated Press. 14,141 articles were collected from Lexis Nexis and Factiva. 2,177 of them were annotated for using economic costs, economic benefits, conservative ideological and uncer- tainty frames. Four SVM models were then trained on 80% of these, one model for each type of frame. The classification tasks were binary, either using or not using the frame. This led to accuracies around 80% across the board. The results of the SVM were then used to quantita- tively analyze the share of stories that featured each frame. This showed that economic cost frames coincided with policy debates, such as COP meetings. There also appeared to be a downward trend for economic cost, against an upward trend for economic benefits. The Wall Street Journal appeared to differ from the other three sources. It featured most economic cost and conservative ideological frames. In general, conservative ideological and uncertainty frames

appeared to be quite uncommon. The latter moreover turned out to be on the decline. Despite the works of Stecula and Merkley [35], Keller et al. [14] and Chinn et al. [11], the use of ML is still rather slim in the field of climate change news. In adjacent climate change fields, there have been works that used machine learning to select scientific publications related to climate change [6, 38, 41] or to automatically categorize government policy docu- ments, e.g. into adaptation-related classes [7, 39, 42]. Employing supervised models to uncover concrete linguistic differences between countries, political orientations or COP meetings is still unexplored. The study at hand considers that analyzing the informative features of classifi- cation and regression algorithms can be a useful contribution to the field. Doing so can help to uncover detailed differences in a large collection of climate change-related articles from a wide range of countries. In sum, there seems to be a gap in the field of climate change news analysis, which machine learning can partly fill up. Manual analysis is currently the dominating approach, yet it suffers from some limitations: it introduces the trade-off between thoroughness and scope and comes with the risks of missing out on important data-ingrained features. Supervised machine learning followed by informative feature analysis can help overcome these limitations. Wherever this technique has been applied so far, it was used chiefly for the prediction of artificial labels, rather than feature analysis. Classification and regression algo-

rithms are capable of processing large amounts of data. More or less independent of the data size, by having specific training objectives, classification models can reveal data patterns that can relatively easily be analyzed. It must be noted that the interpretation of most informative features remains a manual pro- cedure. However, incorporating supervised machine learning into the analysis of the corpus postpones this risk until the very last phase. It also enables analysis of a larger corpus in a con- siderably shorter amount of time and ensures features are processed entirely systematically. This distinguishes feature analysis based on machine learning from manual analysis and makes it a fundamentally different means of studying climate change news than has been used so far. Despite the potential, machine learning is not a flawless method. It should therefore be used to complement human analysis, which is more elaborately addressed under Limitations. The potential of analyzing models in such a way, i.e. extracting the most informative fea- tures, has already been shown in various studies, e.g. on author profiling [12] or political PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 6 / 27

orientation prediction [13] for tweets. However, informative features are regularly seen as a byproduct and are presented in word clouds [13] or plots [12]. It is considered worthwhile to focus more on these informative features and perform a deeper analysis of them. Classification models may represent more general trends than manual analyses can capture and may contain details that human analysts could potentially overlook. This can come to its full potential when systematically collected cross-country data is compared along geographical, political and temporal axes. Data & method the goal of the study was to analyze differences between articles from several countries, politi- cal orientations, and COP meetings. First, a computational data selection strategy was employed to find features inductively at scale. Subsequently, the selected data was preprocessed in order to filter out many overpredic- tive features. To allow the machine learning models to be evaluated on unseen data, separate training and evaluation sets were created. As the last preparation step, features were extracted using the techniques described below. Then, several classification models were trained and a regression model was developed to predict the COP meetings. Finally, the most informative features were analyzed. Data A unique and entirely new dataset was created to analyze international newspaper articles on climate change. The data was collected through Nexis Uni (internationalsales.lexisnexis.com/products/ nexis-uni), the research platform from Lexis Nexis. Under license, this platform provides

access to full-text publications from over 1,000 newspapers. As a first step, a selection of newspapers was made. Only English-language newspapers were selected, to facilitate employment of statistical machine learning algorithms and feature interpretation by the authors. English is the most spoken language across the world and has the third most native speakers (ethnologue.com/insights/most-spoken-language/), guarantee- ing sufficient coverage. The G22 was used to narrow down the scope to sufficiently impactful countries. This group of 22 countries was mostly constituted based on financial impact (imf.org/en/About/ Factsheets/A-Guide-to-Committees-Groups-and-Clubs#GS). Six countries from five conti- nents were chosen: (1) Australia (Australia), (2) Canada (North America), (3) India (Asia), (4) South Africa (Africa), (5) the United Kingdom (Europe) and (6) the United States (North America). This was based on geographical spread, availability of English-language newspapers, and their accessibility through Nexis Uni. However, since the distribution of the data featured several important irregularities (see S1 Appendix), only articles from a subset of countries (Australia, Canada, and the United King- dom) were eventually selected for the analysis. Since these countries are all from different con- tinents, the subset was still suitable to answer the research questions. Each country was represented with newspapers from each end of the political spectrum (e.g. liberal/conservative) to reflect political orientation. The knowledge of the authors was

used to assign these labels, substantiated by analyses from Media Bias Fact Check (mediabiasfactcheck.com) and AllSides (www.allsides.com/media-bias/media-bias-ratings) for the orientation labels. Both sources categorize newspapers using one out of five labels (left, left-center, center, right-center, right). The end result was a selection of 7 newspapers, shown PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 7 / 27

in Table 1. Only newspapers classified as left-center or right-center made it to the final selec- tion. Although the goal was to represent political opposites, the labels from the five-class scale were left intact. One extra left-center newspaper was selected for Australia, considering the right-center The Australian has a substantially larger circulation than any left-center Australian newspaper. As a second step, the search keywords on climate change were determined, considering that not every article that covers the subject literally mentions “climate change”. To generate a broader search query and prevent bias due to human-driven selection, the contents of 230 pages on Wikipedia (en.wikipedia.org) were retrieved and computationally analyzed using tf-idf scores to find climate change keywords. Tf-idf represents an established method for uncovering words that demonstrate relative singularity within a specified subject matter or document [43]. This involved the pages on United Nations Climate Change conference and Global warming, those for each COP meeting, and all pages linked by non-capitalized phrases on the former two pages. By defining tf-idf and document frequency thresholds, the final result was a list of 24 unigrams and 2 bigrams: (1) agreement, (2) air, (3) atmosphere, (4) carbon, (5) carbon dioxide, (6) climate, (7) climate change, (8) co2, (9) dioxide, (10) earth, (11) emissions, (12) gas, (13) global, (14) greenhouse, (15) heat, (16) ice, (17) land, (18) nations, (19) ocean, (20)

protocol, (21) sea, (22) solar, (23) species, (24) surface, (25) temperature and (26) warming. As a third step, the time frames for collection were defined, based on the occurrence of COP meetings. Keller et al. [14] have shown that news publications on climate change peaked around COP meetings. These meetings are (usually) held annually, as the main gathering of the Conference of the Parties (COP) from the United Nations Framework Convention on Cli- mate Change (UNFCCC) (unfccc.int/process/bodies/supreme-bodies/conference-of-the- parties-cop). The time frames around all 26 COP meetings that have taken place between 1995 and 2019 (COP 1 until 25, including the additional COP6a) were included. For each COP meeting, col- lection started one week before the official start and ended one week after the official ending. The average time frame covered 25 days, varying from 23 up to 27 days. The articles were collected on a per-time frame basis. A search string was formed to retrieve full matches of one or multiple keywords, as can be seen in Fig 1 in S2 Appendix. Unfortunately, Nexis Uni did not offer functionality to download more than 100 articles at once and only provided articles as formatted files (PDF). Since each time frame featured thou- sands of search results, collection was automatized using Python, relying on Selenium (selenium-python.readthedocs.io), ChromeDriver (chromedriver.chromium.org) and Chro- mium (chromium.org/Home). Due to hard- and software limitations, only the first 75% of

articles were downloaded for the majority of COPs. With the articles already being sorted by relevance in the search results, this was not considered an issue. Table 1. Overview of countries and newspapers selected for data collection and their share of articles in the dataset. Country Newspaper Political orientation Number of art. Australia The Australian Right-Center 6,547 Sydney Morning Herald Left-Center 5,590 The Age Left-Center 4,321 Canada The Globe and Mail Right-Center 5,556 The Toronto Star Left-Center 4,871 United Kingdom The Times Right-Center 6,770 The Guardian Left-Center 9,917 https://doi.org/10.1371/journal.pone.0297644.t001 PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 8 / 27

The collection took place between 23 July and 4 September 2020, in reverse order of COP meetings, starting with COP25 and ending with COP1. Next, the plain text of each article was extracted from the collected PDFs. This was achieved using Python’s pdftotext (github.com/jalan/pdftotext) library. The result was sanitized by remov- ing (1) unwanted newlines, (2) multiple consecutive spaces and (3) page headings and footers. All elements of the article were separately extracted, being (1) the newspaper (2) the date, (3) the headline, (4) the body text and (5) classifications by Lexis Nexis. This data was stored in 26 JSON files, one per COP meeting. Together, the full dataset for six countries contained 337,777 articles, on average 12,991 per COP meeting. 4,309 PDF articles could not be pro- cessed, 1.3% of all articles. Fig 2 in S2 Appendix shows the distribution of articles over COP meetings. Apparent is the overall increase of article counts over the years. At this point, the dataset contained matches for at least one of the keywords. However, a single occurrence of a keyword (like “ocean”) does not guarantee a relation to climate change. Therefore, a procedure was used to select all articles that contained at least 3 of the selected keywords. The full procedure is explained below. Scatter plots of absolute and unique keyword matches per COP meeting showed that key- word matches were highest for the earlier retrieved articles. Exemplar scatter plots can be seen in Figs 4, 5 in S2 Appendix.

Keller et al. [14] have shown a search strategy with multiple co-occurring keywords is use- ful. Hence, articles that contained less than 3 unique keywords were discarded from the data- set. While this could not guarantee matches solely covering climate change, it was considered that chances increased substantially by applying this approach, without disproportionately reducing the size of the dataset. Filtering by 3 co-occurring keywords reduced the full dataset to 62,824 articles, on average 2,416 per COP. Fig 3 in S2 Appendix visualizes this reduction. A threshold of 4 keywords was also considered, but this was discarded as it reduced the dataset to only 31,650 articles. Eventually, given the irregularities observed in the full dataset (S1 Appendix), only the 43,572 articles from Australia, Canada, and the United Kingdom were used for model develop- ment, shifting the average number of articles per COP to 1,676. More details are specified in Table 7 in S3 Appendix. Three article label categories were then needed: (1) the country of the newspaper, (2) the political orientation of the newspaper, and (3) the COP meeting. The articles were already bundled per COP meeting. The former two labels could directly be derived from the publish- ing newspaper and assessments by Media Bias Fact Check and AllSides. Table 2 shows the label distributions for the eventual dataset. The outstanding Australian coverage observed by Holt and Barkemeyer [16] is not seen here. Nonetheless, the overall

increase over time that Schmidt et al. [17] and Keller et al. [14] described, is apparent in the data- set at hand, which is also clearly visible in Fig 3 in S2 Appendix, which visualizes the full dataset. Preprocessing To further optimize the data for machine learning, extensive preprocessing was employed. This entailed filtering the contents of the articles, rather than discarding articles in their entirety as in previous phases. The preprocessing involved 8 individual filtering steps which were crafted partly during intermediate training runs of the model, as shown in S4 Appendix. Most preprocessing relied entirely on case-insensitive regular expressions to find and remove unwanted phrases. Named entity removal relied on SpaCy’s (spacy.io) tokenization, allowing to filter the data token by token. The tokens were merged into a single string again afterward. PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 9 / 27

Splitting To be able to evaluate the classification and regression models, not all available data was used for training. First, 20% of the data was randomly split off as Test set, i.e. data left unseen by all models and used for the final evaluation. The remaining 80% was further split into a Train set for training the models during experiments and a Dev set for evaluating these experimental models, again randomly and with an 80%-20% ratio. This translates as 64% and 16% of all data respectively. In the end, the combined Train-Dev set was used to train the final model, since the Test set was available for evaluation. For each split, a random seed was used to ensure ran- domness while keeping reproducibility. The label distributions in each subset of the data were verified to be roughly consistent and true to the distribution in the original dataset. Features The main features were word and character n-grams. These were represented by tf-idf rele- vance scores. For each model, tf-idf scores were calculated over the whole corpus using TfidfVectorizer (scikit-learn.org/stable/modules/generated/sklearn.feature_extraction. Table 2. Overview of the distributions for the country and political orientation labels. Country Political orientation COP meeting United Kingdom Australia Canada Left-Center Right-Center Total COP1 227 271 363 616 245 861 COP2 269 371 300 494 446 940 COP3 309 591 653 958 595 1,553 COP4 403 363 552 830 488 1,318 COP5 263 361 412 523 513 1,036 COP6 427 305 350 668 414 1,082 COP6a 351 468 415 667 567 1,234

COP7 344 467 453 710 554 1,264 COP8 288 360 374 546 476 1,022 COP9 364 437 340 640 501 1,141 COP10 258 392 289 535 404 939 COP11 558 545 442 826 719 1,545 COP12 673 1,087 469 1,209 1,020 2,229 COP13 758 1,045 615 1,380 1,038 2,418 COP14 575 968 391 1,037 897 1,934 COP15 1,136 1,531 517 1,642 1,542 3,184 COP16 715 814 331 1,015 845 1,860 COP17 718 840 402 1,087 873 1,960 COP18 797 663 323 942 841 1,783 COP19 825 710 308 1,115 728 1,843 COP20 981 629 350 1,283 677 1,960 COP21 1,695 866 507 2,080 988 3,068 COP22 752 522 290 827 737 1,564 COP23 626 549 268 759 684 1,443 COP24 1,107 627 352 1,036 1,050 2,086 COP25 1,268 676 361 1,274 1,031 2,305 Average 642 633 401 950 726 1,676 Total 16,687 16,458 10,427 24,699 18,873 43,572 https://doi.org/10.1371/journal.pone.0297644.t002 PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 10 / 27

text.TfidfVectorizer) from Scikit Learn. The n-gram ranges could be of different lengths, so that not only word unigrams, but also bi- and trigrams, as well as character multigrams could be included by choice during experiments. The headline and the body text of each article were handled separately, allowing the use of different settings for each element. Classification Architecture. Classification was implemented using Scikit Learn’s LinearSVC (scikit- learn.org/stable/modules/generated/sklearn.svm.LinearSVC) class. This features a linear implementation of support vector machines (scikit-learn.org/stable/modules/svm) (SVM), which is a popular and effective architecture in natural language processing. The default set- tings for LinearSVC were used so that the focus could be on feature generation. For each of the three labels (countries, political orientations and COP meetings) an inde- pendent classification model was trained. All three models were trained consecutively during experiments with different features and settings. This allowed to constantly evaluate the impact of changes across aspects. During experiments, the models were trained in a single run on the Train data and consec- utively evaluated on the Dev set. Eventually, a final model was trained using the optimal con- figuration on the Train-Dev data and evaluated on the so far unseen Test set. Performance of each created model was evaluated by means of Precision, Recall and F1-score. The goal was to determine and analyze the most informative features, i.e. the textual fea-

tures that contribute most to the model’s decision (as done before, for example, by [12, 13]). This is considered an objective way to determine which terms are characteristic or rather uncharacteristic for each country, political orientation and COP meeting. In a broader sense, this allows to uncover topical and terminological differences between the articles for these labels. While this analysis is purely textual and does not consider context or meaning, it can still provide insight into a large dataset in a neutral manner. Concretely, in order to assess the contribution of each single feature, the model’s coefficients were extracted. In Scikit Learn, coefficents can be easily accessed using LinearSVC.coef_. A more detailed explanation can be found on scikit-learn.org/stable/modules/svm.html. Both positive and negative coefficients are generated for each class. These are the weights of the fea- tures, with absolutely higher weights indicating a stronger contribution of the corresponding feature to the models’ decision. Since a lot of features were generated, only the ones with the most positive and negative coefficients were presented. For each given class (for example the country “Australia” in coun- try detection), features with positive weights can be seen as strongly characteristic for that class, and ones with negative weights as uncharacteristic. This allowed to analyze the textual characteristics of each class. To provide more perspective for the scores that would be yielded by the SVM, a baseline

was first generated using the DummyClassifier (scikit-learn.org/stable/modules/ generated/sklearn.dummy.DummyClassifier) class. A stratified strategy was employed, which leads to random predictions solely based on the class distribution within the training data. This can be interpreted as educated guessing. The SVM models should thus at least outperform this baseline to be considered useful. When trained on the Train-Dev set and evaluated on the Test data, the baseline yields a macro F1-score of 0.33 for countries, 0.51 for political orientations and 0.03 for COP meetings. These scores are as expected for random guessing with 3, 2 and 26 classes respectively. Experiments. Several experiments were conducted to determine the most useful combi- nation of features and preprocessing steps. This assessment was based both on performance PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 11 / 27

scores and most informative features. The majority of experiments were executed on the Pere- grine HPC cluster of the University of Groningen, and some additional runs were performed on Hábrók, its successor (rug.nl/society-business/centre-for-information-technology/ research/services/hpc/facilities/peregrine-hpc-cluster). The accuracy and macro F1-scores yielded by all 18 experiments can be seen in Table 8 in S5 Appendix. These models were trained on Train data and evaluated on the Dev set. For some final experiments, word embeddings were incorporated into the feature set. While this type of feature is not useful in answering the research questions at hand, embed- dings have proven to be powerful predictors in text classification [44]. Therefore, the impact of implementing them (or conversely, avoiding their use) was evaluated. The 300-dimensional word embeddings vectors from Google’s word2vec project (code.google.com/archive/p/ word2vec/) were used here as averaged document vectors. Performance improved only slightly or not at all when these vectors were added to the feature set, which justified the use of solely textual features for further model development, given that explainability was favored over per- formance in this context. Taking all experiments into account, the configuration with word 1–3-grams and the full preprocessing strategy (Experiment 15) was chosen as the final configuration for all three aspects. Tables 3–5 show the classification reports for the final model. This was trained using

this optimal configuration on Train-Dev data, and then evaluated on the Test set. This led to scores that were only slightly deviant from those in the experimental phase, where training was performed on Train and evaluation on Test data. Regression Architecture. Apart from classification, an implementation of regression was developed for the COP meeting labels. These labels are numeric and have a temporal nature. Thus, it is better to predict a meeting closer to the real label than one that is several years off. This is not taken into account when using classification with F1-scores for evaluation, where a prediction can be either wrong or right. A regression model is therefore considered a more suitable way to predict the COP label. This type of model leads to a continuous number and uses the differ- ence from the correct value for evaluation, called the error. Regression was implemented using the LinearRegression (scikit-learn.org/stable/ modules/generated/sklearn.linear_model.LinearRegression) class from Scikit Learn. This was Table 3. Classification report of the final model for country labels. precision recall f1-score Australia 0.89 0.89 0.89 Canada 0.94 0.88 0.91 United Kingdom 0.88 0.92 0.90 Macro average 0.90 0.89 0.90 https://doi.org/10.1371/journal.pone.0297644.t003 Table 4. Classification report of the final model for political orientation labels. precision recall f1-score left-center 0.79 0.90 0.84 right-center 0.84 0.68 0.75 Macro average 0.81 0.79 0.80 https://doi.org/10.1371/journal.pone.0297644.t004 PLOS ONE

Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 12 / 27

integrated in the existing pipeline and the approach to preprocessing, splitting the data and generating features remained the same consequently. The COP meeting labels were converted to numbers. COP6a was represented as 6.5 and all other meetings as their round edition number. The models were then trained using similar n- gram feature combinations as in the classification experiments. The performance of the resulting models was evaluated using the Mean Absolute Error (MAE), Mean Squared Error (MSE) and Root Mean Squared Error (RMSE). Like for classification, the most informative features were extracted by linking the model’s coefficients to the original features. For their nature to predict continuous numbers, regression models only yield overall coefficients. These are less insightful than the per-class coefficients from classification. However, the coefficients do indicate what drew the model towards a higher or a lower prediction. Unlike the classification model, no baseline was defined for regression. A rule-based regres- sor like DummyRegressor (scikit-learn.org/stable/modules/generated/sklearn.dummy. DummyRegressor) relies on always predicting a single value, such as the mean or the median of the labels. Having as many as 26 COP meetings that moreover have a temporal nature, such a model would inevitably produce high errors that are effectively meaningless. Table 5. Classification report of the final model for COP meeting labels. precision recall f1-score COP1 0.58 0.37 0.45 COP2 0.46 0.30 0.36 COP3 0.41 0.52 0.46 COP4 0.51 0.49 0.50

COP5 0.34 0.24 0.28 COP6 0.38 0.23 0.28 COP6a 0.41 0.35 0.38 COP7 0.41 0.39 0.40 COP8 0.56 0.23 0.32 COP9 0.25 0.22 0.23 COP10 0.33 0.13 0.18 COP11 0.43 0.27 0.33 COP12 0.39 0.50 0.44 COP13 0.42 0.49 0.45 COP14 0.46 0.47 0.47 COP15 0.50 0.66 0.57 COP16 0.51 0.34 0.41 COP17 0.42 0.43 0.42 COP18 0.56 0.41 0.47 COP19 0.48 0.49 0.49 COP20 0.54 0.49 0.51 COP21 0.46 0.76 0.58 COP22 0.53 0.49 0.51 COP23 0.63 0.42 0.50 COP24 0.59 0.59 0.59 COP25 0.53 0.74 0.62 Macro avg 0.46 0.42 0.43 https://doi.org/10.1371/journal.pone.0297644.t005 PLOS ONE Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 13 / 27

Experiments. To determine an optimal configuration for the regression algorithm, a cou- ple of experiments were conducted. Since findings similar to those in the classification experi- ments were expected, the intermediate pre-processing strategies were not tested with the regression algorithm. The performance of the models was compared by use of all three error metrics, being MAE, MSE and RMSE. The error scores and R2 for the final model using the opti- mal configuration are in Table 6. Table 9 in S5 Appendix shows the scores from all experiments. Reproducibility All code employed, plots generated and most informative features yielded in this study are available on DataverseNL: doi.org/10.34894/AMISPR. The dataset was collected from Nexis Uni under license of the University of Groningen, and is therefore not publicly available. The data can be reproduced by licensees using the provided scripts. Results The most informative features yielded by the three classification models (COPs, countries and political orientations) and the regression model for COPs were analyzed to uncover what each model had learned from the data. The most positive and negative features for each class were listed, with the extremest coeffi- cients first. Herein, the positive features can be seen as characteristic of that class, and the nega- tive ones as uncharacteristic. For the COP classification model, this was limited to 50 features from each side of the spec- trum. A limit of 100 was applied for the political orientation and country models. This means

that 2,600, 600, and 200 most informative features were extracted from the three models respectively. The 250 most positive and negative informative features from the COP regression model were extracted, i.e. 500 in total. Seeing that a regression model only yields a single set of coeffi- cients, inherently not structured by class, it was useful to extract more features. These features were manually analyzed for topical and terminological differences. This was largely a qualitative analysis of the n-grams, subject to the scope of expertise and knowledge of the study’s authors. Some quantitative aspects were considered in this analysis as well, such as the height of the corresponding coefficient and the overlap of features across classes. For regres- sion, coefficient height played a larger role, as to comprehend the single set of features yielded. The analysis below provides an exploration of the features aiming to answer the research questions and assess the potential of the methodology, including finding room for future improvements. Apart from the analysis, the full lists of most informative features must in itself be seen as an important result of this study. These can be accessed through OSF as found under Repro- ducibility. Extended details about the performance of the classification and regression models can be found under Method in S5 Appendix. Table 6. Error scores of the final regression model for COP meeting numbers. MAE MSE RMSE R2 COP meeting 2.95 16.59 4.07 0.65 https://doi.org/10.1371/journal.pone.0297644.t006 PLOS ONE

Learning from climate change news PLOS ONE | https://doi.org/10.1371/journal.pone.0297644 March 20, 2024 14 / 27