How relevant is climate change research for climate change policy? An empirical analysis based on Overton data

documents than expected given their overall share of policy documents (this result may be partly driven by the biased coverage of the Overton database). The policy documents published by the different types of institutions have especially cited more recent publications. Since cli- mate change is of great societal relevance worldwide, research activities are on a high level (compared to other topics) that can be picked up in a timely manner by the policy area. Although one might expect that policy and science impact correlate (what is relevant for the scientific discourse might be equally relevant for the policy discourse), we found the opposite: The correlation between policy citations and science citations and the correlation between pol- icy citations and the impact factor of the journals publishing the papers are both low. Thus, it seems that both areas of society (science and policy) focus on different papers from climate change research. If the scientific discourse and the policy discourse are scarcely related in terms of citation counts, one might expect that they focus on different fields. Our results reveal, however, that this is not the case: Climate change papers with at least one policy citation are concentrated on similar fields as all climate change papers (biology, earth sciences, engineer- ing, and disease sciences). Since field differences scarcely exist between both publication sets of interest, it would be interesting to explore in future studies how the differences can be charac- terized by other means.

What are the policy sources that are very active in the political climate change discourse and which scientific institutions provide the necessary scientific information? Our results show that the Publication Offices of the European Union, World Health Organization, and World Bank have published the most climate change policy documents. Since climate change is a worldwide problem and demand, it comes as no surprise that these cross-regional institu- tions have the highest publication output. The relevant science institutions for policy sources are mostly institutions with high reputation in science–this might be in contrast to the low cor- relation between science and policy citations on the single paper level. On the institutional level, policy sources seem to trust scientific institutions being renowned for reputable research on climate change (e.g., the University of East Anglia). In this study, we found that some research outcomes seem to be more relevant for the scien- tific discourse and some outcomes that seem to be more relevant for the policy area. This dis- crepancy has been found also in other studies. One reason for the differences might be barriers to academic outcomes from policy institutions such as access to climate change publications [18]. Another reason might be missing summaries of research results that are understandable for people outside academia. Bornmann and Marx [47] recommend therefore that researchers should write assessment reports (such as the IPCC) summarizing “the status of the research on

a certain subject . . . Societal impact is given when the content of a report is addressed outside of science (in a government document, for example)” (p. 211). Our analyses revealed the challenges in measuring policy impact via citation patterns. In fact, the closer a document is related to actual decision-making the fewer citations it may con- tain. For example, scientific assessments of the literature contain large numbers of citations, but they are not directly used in policy-making. Instead they are further built upon and “trans- lated” in policy briefs, policy reports, briefing notes or ministerial expertise. The final political decision–usually a legal text–usually does not contain any citations. As we move towards real decisions it therefore gets increasingly challenging to measure impact in this way. Future work may therefore be organized around a simple model of policy impact considering a chain of dif- ferent document types. Scientific assessment reports, systematic reviews or meta-analyses–as recommended by Bornmann and Marx [47]–may be the starting point as rigorous syntheses of the available summaries. Next might be science communication documents such as policy briefs, policy reports or plain language summaries. Government reports might be compiled to directly inform particular decisions and, finally, legislative documents cover the policies PLOS ONE How relevant is climate change research for climate change policy? PLOS ONE | https://doi.org/10.1371/journal.pone.0274693 September 22, 2022 21 / 24