How relevant is climate change research for climate change policy? An empirical analysis based on Overton data

papers in this area have limited policy impact, but some papers such as the papers selected for their case study received a lot of policy impact. Hicks and Isett [25] explain in detail the policy impact of the selected papers. For example, the authors selected the well-known study by Mans- field [26], Mansfield [27] that estimated the social rate of return to public research spending. Hicks and Isett [25] describe the diverse policy impact reached by this paper using several sources. In the most recent study, Pinheiro, Vignola-Gagne´ [28] used publication data from Frame- work Programmes (FPs) for Research and Technological Development. The authors investi- gated the relationship of cross-disciplinarity on the paper level and policy impact measured by policy citation data from the Overton database. Pinheiro, Vignola-Gagne´ [28] conclude as fol- lows: “Our approach enables testing in a general way the assumption underlying many funding programs, namely that cross-disciplinary research will increase the policy relevance of research outcomes. Findings suggest that research assessments could benefit from measuring uptake in policy-related literature, following additional characterization of the Overton database; of the science-policy interactions it captures; and of the contribution of these interactions within the larger policymaking process” (p. 616). Dataset used For many years, policy documents’ and policy citations’ data were aggregated only by the com- panies Altmetric and PlumX. Recently, however, the Overton database (see https://www.

overton.io) was launched with the goal of becoming the largest database of policy documents and citations [29]. In Overton, policy documents are defined “very broadly as documents writ- ten primarily for or by policymakers” (see http://help.overton.io/en/articles/3823271-what-s- your-definition-of-a-policy-document). Overton includes documents from governments, think tanks (i.e., research institutions that perform research and advocacy in climate change), non-governmental organizations (NGOs) and intergovernmental organizations (IGOs, i.e., organizations that are composed of states) (see http://help.overton.io/en/articles/ 5062448-which-publications-does-overton-collect). The database includes not only various bibliographic information on policy documents (e.g., title and appearance), but also the cita- tion links that exist between policy and science as well as among the policy documents in the database themselves. The citation relations are identified by Overton by using text-mining methods. According to Yin, Gao [11], the Overton database “includes all major economies and large population centers, with a notable exception of mainland China” (p. 128). The data- base is updated on a weekly basis. In December 2020, the database includes 799,716 policy doc- uments with citation relations to either other policy documents or scientific papers in 66 different languages from 168 countries (including the European Union and IGOs) and more than 1250 different policy sources. Yin, Gao [11] studied the reliability of the science-policy citations in the Overton database,

by comparing them with the citation links provided by the Microsoft Academic Graph data- base (see https://academic.microsoft.com/home). The results show that “although the two datasets are collected for different purposes using different approaches and technologies, the measurements carried out independently across the two datasets show remarkable consisten- cies” (p. SI). Since the results by Yin, Gao [11] confirm the reliability of the Overton data, we decided to use the data for the current study on climate change. Overton provided a snapshot (dated December 04, 2020) of their database to some of us (LB and RH). This snapshot has been imported into a local PostgreSQL database at the Max Planck Institute for Solid State Research (Stuttgart, Germany). After an analysis of publication dates of policy documents and consultation with Euan Adie (Overton), we excluded the policy documents with the publica- tion dates ‘1970-01-01’, ‘1970-01-02’, and ‘2002-07-01’ from our analysis because they were PLOS ONE How relevant is climate change research for climate change policy? PLOS ONE | https://doi.org/10.1371/journal.pone.0274693 September 22, 2022 5 / 24