How relevant is climate change research for climate change policy? An empirical analysis based on Overton data

or challenge an idea) among others, suggesting the need to understand the semantics of the policy science citations” (p. SI). This study focusses on the connection of climate change research and policy. The study is based on data from the (new) Overton database including policy documents (10,846 climate change policy documents covered in the database) and their citations of scientific publications. With this study, we followed other studies using Overton data investigating links between pol- icy and research (e.g., on COVID-19). Although the Overton database captures a large collec- tion of policy documents, potential biases in coverage and data sample cannot be excluded [11]. For example, the Overton providers will not have access to many governmental archives, and if they have access, it will be restricted to only a part of the existing documents. Other shortcomings of Overton are mentioned by Yang, Huang [9]: “the metadata of such policy documents cannot reveal the semantic information contained in the policy process. At the same time, some policy documents have unstructured features, so attribute identification and labeling may be required”. Overton uses a very broad definition of policy documents, i.e., “documents primarily written by and for policy makers”. The idea behind this is to cover not only text that documents the pol- icy or legislation itself in the corpus, but also documents that were written to inform or influ- ence decisions. Our analyses do not distinguish between those two fundamentally different

classes of policy documents. Documents written for policymakers are often written with the purpose to inform or influence documents authored by policymakers and are as such funda- mentally different from documents authored by scientists. Moreover, under this wide umbrella definition there are very different types of documents: scientific assessments by the scientific community, legislations, policy reports by IGOs and NGOs, policy briefs, speeches etc. The different nature of these documents explains some of the results here. For example, it is the main purpose of scientific assessments as those by IPCC to assess the state of knowledge in climate change research and inform international climate diplomacy and national climate pol- icy with robust evidence. In nature, these assessments are comprehensive reviews of the litera- ture with tens of thousands of references. On the other hand, policy briefs are designed for communications and often deliberately strip out literature sources. The policy impact analysis in this study, therefore to some extent simply highlights different policy document types. Any interpretation of policy impact of research can only be undertaken based on such an important caveat. In this study, we empirically targeted several aspects of the connection between climate change research and policy. Focusing on the time trend of this connection reveals that the dis- cussion of climate change in policy seems to have had its peak some years ago. Although we suspect a lot of uncertainty related to the coverage of policy documents in Overton, there

seems to be an impact of international climate policy cycles on policy document publication. We observe local peaks in climate policy documents around major decisions in international climate diplomacy. For example, we observe temporal peaks in policy documents around the failed Copenhagen Summit in 2009 and the Paris Agreement; there is a growth in policy docu- ments from IPCC’s Fifth Assessment in 2013/2014 with a peak in 2015 when the Paris Agree- ment was made. IPCC reports might play a particular role as they are usually released 2–3 years ahead of major international climate diplomacy events and could trigger substantial co- publication activities. In 2023, the first Global Stocktake on progress with the Paris Agreement is scheduled with IPCC AR6 being released during 2021 and 2022. We might thus expect to see increases in climate change policy documents and citations to the scientific literature in the 2–3 years following. Various types of institutions publish policy documents. Our results point out that IGOs and think tanks–with a focus on climate change–have published more climate change policy PLOS ONE How relevant is climate change research for climate change policy? PLOS ONE | https://doi.org/10.1371/journal.pone.0274693 September 22, 2022 20 / 24

documents than expected given their overall share of policy documents (this result may be partly driven by the biased coverage of the Overton database). The policy documents published by the different types of institutions have especially cited more recent publications. Since cli- mate change is of great societal relevance worldwide, research activities are on a high level (compared to other topics) that can be picked up in a timely manner by the policy area. Although one might expect that policy and science impact correlate (what is relevant for the scientific discourse might be equally relevant for the policy discourse), we found the opposite: The correlation between policy citations and science citations and the correlation between pol- icy citations and the impact factor of the journals publishing the papers are both low. Thus, it seems that both areas of society (science and policy) focus on different papers from climate change research. If the scientific discourse and the policy discourse are scarcely related in terms of citation counts, one might expect that they focus on different fields. Our results reveal, however, that this is not the case: Climate change papers with at least one policy citation are concentrated on similar fields as all climate change papers (biology, earth sciences, engineer- ing, and disease sciences). Since field differences scarcely exist between both publication sets of interest, it would be interesting to explore in future studies how the differences can be charac- terized by other means.

What are the policy sources that are very active in the political climate change discourse and which scientific institutions provide the necessary scientific information? Our results show that the Publication Offices of the European Union, World Health Organization, and World Bank have published the most climate change policy documents. Since climate change is a worldwide problem and demand, it comes as no surprise that these cross-regional institu- tions have the highest publication output. The relevant science institutions for policy sources are mostly institutions with high reputation in science–this might be in contrast to the low cor- relation between science and policy citations on the single paper level. On the institutional level, policy sources seem to trust scientific institutions being renowned for reputable research on climate change (e.g., the University of East Anglia). In this study, we found that some research outcomes seem to be more relevant for the scien- tific discourse and some outcomes that seem to be more relevant for the policy area. This dis- crepancy has been found also in other studies. One reason for the differences might be barriers to academic outcomes from policy institutions such as access to climate change publications [18]. Another reason might be missing summaries of research results that are understandable for people outside academia. Bornmann and Marx [47] recommend therefore that researchers should write assessment reports (such as the IPCC) summarizing “the status of the research on

a certain subject . . . Societal impact is given when the content of a report is addressed outside of science (in a government document, for example)” (p. 211). Our analyses revealed the challenges in measuring policy impact via citation patterns. In fact, the closer a document is related to actual decision-making the fewer citations it may con- tain. For example, scientific assessments of the literature contain large numbers of citations, but they are not directly used in policy-making. Instead they are further built upon and “trans- lated” in policy briefs, policy reports, briefing notes or ministerial expertise. The final political decision–usually a legal text–usually does not contain any citations. As we move towards real decisions it therefore gets increasingly challenging to measure impact in this way. Future work may therefore be organized around a simple model of policy impact considering a chain of dif- ferent document types. Scientific assessment reports, systematic reviews or meta-analyses–as recommended by Bornmann and Marx [47]–may be the starting point as rigorous syntheses of the available summaries. Next might be science communication documents such as policy briefs, policy reports or plain language summaries. Government reports might be compiled to directly inform particular decisions and, finally, legislative documents cover the policies PLOS ONE How relevant is climate change research for climate change policy? PLOS ONE | https://doi.org/10.1371/journal.pone.0274693 September 22, 2022 21 / 24

themselves. In this context, Isett and Hicks [48] speak about knowledge intermediaries in doc- ument chains. Future research could attempt measuring the impact on policy along such a document chain. As citations would be expected to fade away as you move down the chain, it will become increasingly relevant to use text mining or other methods from natural language processing (e.g., text similarity approaches; argumentation mining) to measure impact. Finally, as primary studies are very dependent on their specific research design, data and methods applied, there is a widespread argument that policy should be informed by the most robust scientific evidence and as such be built from secondary research (reviews) whenever possible [49]. Therefore, future scientometric research may explore to what extent primary and secondary research is used in policy documents and how this varies across different sectors. Acknowledgments The bibliometric data used in this paper are from an in-house database developed and main- tained by SciTech Strategies, Inc. derived from Scopus, prepared by Elsevier BV (Amsterdam, The Netherlands). The policy document data were shared with us by Overton on December 04, 2020. Author Contributions Conceptualization: Lutz Bornmann, Robin Haunschild, Werner Marx, Jan C. Minx. Data curation: Robin Haunschild, Kevin Boyack. Formal analysis: Robin Haunschild, Kevin Boyack. Investigation: Robin Haunschild, Kevin Boyack. Methodology: Lutz Bornmann, Robin Haunschild, Kevin Boyack. Supervision: Jan C. Minx. Visualization: Kevin Boyack.

Writing – original draft: Lutz Bornmann, Robin Haunschild, Kevin Boyack, Werner Marx, Jan C. Minx. Writing – review & editing: Lutz Bornmann, Robin Haunschild, Kevin Boyack, Werner Marx, Jan C. Minx. References 1. Arrhenius S. On the influence of carbonic acid in the air upon the temperature of the ground. Philosophi- cal Magazine and Journal of Science Series. 1896; 5(41):237–76. 2. Weart SR. The discovery of global warming. Cambridge, MA, USA: Harvard University Press; 2008. 3. IPCC. Climate change 2014: Synthesis report. Contribution of working groups I, II and III to the fifth assessment report of the Intergovernmental Panel on Climate Change [core writing team, R.K. Pachauri and L.A. Meyer (eds.)]. Geneva, Switzerland: IPCC, 2014. 4. IPCC. Climate change 2021: The physical science basis. Contribution of working group I to the sixth assessment report of the Intergovernmental Panel on Climate Change [Masson-Delmotte V., Zhai P., Pirani A., n, Pe´ an C., Berger S., Caud N., Chen Y., Goldfarb L., Gomis M. I., Huang M., Leitzell K., Lon- noy E., Matthews J. B. R., Maycock T. K., Waterfield T., Yelekc¸i O., Yu R., and o B.(eds.)]. Cambridge, UK: Cambridge University Press, 2021. 5. Edenhofer O, Minx J. Mapmakers and navigators, facts and values. Science. 2014; 345(6192):37–8. https://doi.org/10.1126/science.1255998 PMID: 24994640 6. Bolin B. A history of the science and politics of climate change: The role of the Intergovernmental Panel on Climate Change. Cambridge, UK: Cambridge University Press; 2007. PLOS ONE

How relevant is climate change research for climate change policy? PLOS ONE | https://doi.org/10.1371/journal.pone.0274693 September 22, 2022 22 / 24

7. Mach KJ, Freeman PT, Mastrandrea MD, Field CB. A multistage crucible of revision and approval shapes IPCC policymaker summaries. Science Advances. 2016; 2(8). https://doi.org/10.1126/sciadv. 1600421 WOS:000383734300017. PMID: 27532046 8. Field CB, Barros VR. Added value from IPCC approval sessions. Science. 2015; 350(6256):36. https:// doi.org/10.1126/science.aaa8976 WOS:000362098300031. PMID: 26430105 9. Yang C, Huang C, Su J. A bibliometrics-based research framework for exploring policy evolution: A case study of China’s information technology policies. Technological Forecasting and Social Change. 2020;157. https://doi.org/10.1016/j.techfore.2020.120116 WOS:000540275500034. 10. Callaghan M, Schleussner CF, Nath S, Lejeune Q, Knutson TR, Reichstein M, et al. Machine-learning- based evidence and attribution mapping of 100,000 climate impact studies. Nat Clim Change. 2021. https://doi.org/10.1038/s41558-021-01168-6 WOS:000706044400005. 11. Yin Y, Gao J, Jones BF, Wang DS. Coevolution of policy and science during the pandemic. Science. 2021; 371(6525):128–30. https://doi.org/10.1126/science.abe3084 WOS:000606520400023. PMID: 33414211 12. Bornmann L. Scientific revolution in scientometrics: The broadening of impact from citation to societal. In: Sugimoto CR, editor. Theories of informetrics and scholarly communication. Berlin, Germany: De Gruyter; 2016. p. 347–59. 13. Tahamtan I, Bornmann L. Altmetrics and societal impact measurements: Match or mismatch? A litera- ture review. El Profesional de la Informacio´ n. 2020; 29(1):e290102.

14. Fang Z, Dudek J, Noyons E, Costas R. New possibilities of studying the interactions between policy and science: A descriptive exploration of the policy documents indexed by Overton. Quantitative Science Studies. in press. 15. Black N, Donald A. Evidence based policy: Proceed with care. British Medical Journal. 2001; 323 (7307):275–9. https://doi.org/10.1136/bmj.323.7307.275 PMID: 11485961 16. Pedersen DB. The political epistemology of science-based policy-making. Society. 2014; 51(5):547–51. https://doi.org/10.1007/s12115-014-9820-z 17. OPENing UP. Deliverable D5.1 –Altmetrics Status Quo. OPENing UP new methods, indicators and tools for peer review, impact measurement and dissemination of research results. Project acronym: OpenUP. Brussels, Belgium: European Commission, 2016. 18. Vilkins S, Grant WJ. Types of evidence cited in Australian Government publications. Scientometrics. 2017; 113(3):1681–95. https://doi.org/10.1007/s11192-017-2544-2 19. Arthur T. Categorizing policy document citations in PlumX 2016 [cited 2017 March 10]. Available from: http://plumanalytics.com/categorizing-policy-document-citations-in-plumx/. 20. Michalek A, Crosby T, Arthur T, Parkhill M, James C, McCullough R, et al. Research assessment met- rics: Past, present and future. Amsterdam, the Netherland: Elsevier; 2017. 21. Bornmann L, Haunschild R, Marx W. Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents? Scientometrics. 2016; 109

(3):1477–95. https://doi.org/10.1007/s11192-016-2115-y WOS:000389336100004. PMID: 27942080 22. Haunschild R, Bornmann L. How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics. 2017; 110 (3):1209–16. https://doi.org/10.1007/s11192-016-2237-2 WOS:000394339500007. PMID: 28255186 23. Tattersall A, Carroll C. What can Altmetric.com tell us about policy citations of research? An analysis of Altmetric.com data for research articles from the University of Sheffield. Frontiers in Research Metrics and Analytics. 2018; 2(9). https://doi.org/10.3389/frma.2017.00009 24. Fang Z, Costas R, Tian W, Wang X, Wouters P. An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics. Scientometrics. 2020; 124 (3):2519–49. https://doi.org/10.1007/s11192-020-03564-9 PMID: 32836523 25. Hicks D, Isett KR. Powerful numbers: Exemplary quantitative studies of science that had policy impact. Quantitative Science Studies. 2020; 1(3):969–82. https://doi.org/10.1162/qss_a_00060 26. Mansfield E. Academic research and industrial innovation. Research Policy. 1991; 20(1):1–12. https:// doi.org/10.1016/0048-7333(91)90080-a WOS:A1991FG31100001. 27. Mansfield E. Academic research and industrial innovation: An update of empirical findings. Research Policy. 1998; 26(7–8):773–6. https://doi.org/10.1016/s0048-7333(97)00043-7 WOS:000073600200004.

28. Pinheiro H, Vignola-Gagne´ E, Campbell D. A large-scale validation of the relationship between cross- disciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database. Quantitative Science Studies. 2021; 2(2):616–42. https://doi.org/10.1162/qss_a_00137 PLOS ONE How relevant is climate change research for climate change policy? PLOS ONE | https://doi.org/10.1371/journal.pone.0274693 September 22, 2022 23 / 24

29. Szomszor M, Adie E. Overton: A bibliometric database of policy document citations 2022 [cited 2022 March 8]. Available from: https://arxiv.org/abs/2201.07643. 30. R Core Team. R: A language and environment for statistical computing. 3.6.0 ed. Vienna, Austria: R Foundation for Statistical Computing; 2019. 31. Wickham H. Tidyverse: Easily install and load the ’Tidyverse’. R package version 1.2.1. 2017 [22 June 2020]. Available from: https://CRAN.R-project.org/package=tidyverse. 32. Teixeira da Silva JA. CiteScore: Advances, evolution, applications, and limitations. Publishing Research Quarterly. 2020; 36(3):459–68. https://doi.org/10.1007/s12109-020-09736-y 33. Lamb WF, Wiedmann T, Pongratz J, Andrew R, Crippa M, Olivier JGJ, et al. A review of trends and driv- ers of greenhouse gas emissions by sector from 1990 to 2018. Environmental Research Letters. 2021; 16(7). https://doi.org/10.1088/1748-9326/abee4e WOS:000667941000001. 34. Minx JC, Lamb WF, Andrew RM, Canadell JG, Crippa M, Do¨ bbeling N, et al. A comprehensive and syn- thetic dataset for global, regional, and national greenhouse gas emissions by sector 1970–2018 with an extension to 2019. Earth System Science Data. 2021; 13(11):5213–52. https://doi.org/10.5194/essd- 13-5213-2021 35. Friedlingstein P, Jones MW, O’Sullivan M, Andrew RM, Bakker DCE, Hauck J, et al. Global carbon bud- get 2021. Earth System Science Data Discussion. 2021; 2021:1–191. https://doi.org/10.5194/essd- 2021-386 36. United Nations Environment Programme. Emissions gap report 2021: The heat is on–A world of climate

promises not yet delivered. Nairobi: United Nations Environment Programme (UNEP) and UNEP DTU Partnership, 2021. 37. Ou Y, Iyer G, Clarke L, Edmonds J, Fawcett AA, Hultman N, et al. Can updated climate pledges limit warming well below 2 degrees C? Increased ambition and implementation are essential. Science. 2021; 374(6568):693–5. https://doi.org/10.1126/science.abl8976 WOS:000714952000018. PMID: 34735225 38. Haunschild R, Bornmann L, Marx W. Climate change research in view of bibliometrics. Plos One. 2016; 11(7). https://doi.org/10.1371/journal.pone.0160393 WOS:000381516300048. PMID: 27472663 39. Berrang-Ford L, Sietsma AJ, Callaghan M, Minx JC, Scheelbeek PFD, Haddaway NR, et al. Systematic mapping of global research on climate and health: A machine learning review. Lancet Planet Health. 2021; 5(8):E514–E25. https://doi.org/10.1016/S2542-5196(21)00179-0 WOS:000685102900011. PMID: 34270917 40. Boyack KW, Klavans R. Creation and analysis of large-scale bibliometric networks. In: Gla¨nzel W, Moed HF, Schmoch U, Thelwall M, editors. Springer handbook of science and technology indicators. Cham, Switzerland: Springer International Publishing; 2019. p. 187–212. 41. Traag VA, Waltman L, van Eck NJ. From Louvain to Leiden: Guaranteeing well-connected communi- ties. Scientific Reports. 2019; 9(1):5233. https://doi.org/10.1038/s41598-019-41695-z PMID: 30914743 42. Martin S, Brown WM, Klavans R, Boyack K, editors. OpenOrd: An open-source toolbox for large graph layout. SPIE 7868, Visualization and Data Analysis; 2011; San Francisco Airport, CA, USA: SPIE.

43. Boyack KW, Newman D, Duhon RJ, Klavans R, Patek M, Biberstine JR, et al. Clustering more than two million biomedical publications: Comparing the accuracies of nine text-based similarity approaches. PLoS ONE. 2011; 6(3):-. https://doi.org/10.1371/journal.pone.0018029 ISI:000288514600022. PMID: 21437291 44. Sparck Jones K, Walker S, Robertson SE. A probabilistic model of information retrieval: Development and comparative experiments: Part 1. Information Processing & Management. 2000; 36(6):779–808. https://doi.org/10.1016/S0306-4573(00)00015-7 45. Sparck Jones K, Walker S, Robertson SE. A probabilistic model of information retrieval: Development and comparative experiments: Part 2. Information Processing & Management. 2000; 36(6):809–40. https://doi.org/10.1016/S0306-4573(00)00016-9 46. Bo¨ rner K, Klavans R, Patek M, Zoss AM, Biberstine JR, Light RP, et al. Design and update of a classifi- cation system: The UCSD map of science. PLOS ONE. 2012; 7(7):e39464. https://doi.org/10.1371/ journal.pone.0039464 PMID: 22808037 47. Bornmann L, Marx W. How should the societal impact of research be generated and measured? A pro- posal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics. 2014; 98(1):211–9. WOS:000329319200015. 48. Isett KR, Hicks D. Pathways from research into public decision making: Intermediaries as the third com- munity. Perspectives on Public Management and Governance. 2019; 3(1):45–58. https://doi.org/10. 1093/ppmgov/gvz020