How relevant is climate change research for climate change policy? An empirical analysis based on Overton data

their relationships. These studies focused on policymaking organizations’ networks, public ser- vice organizations’ networks, and policy collaboration networks. In the area of altmetrics research, a recent overview of studies on measuring policy impact using altmetric data can be found in Fang, Dudek [14] and Yang, Huang [9]. A number of studies has used policy impact data from Altmetric (https://www.altmetric.com) or PlumX (https://plumanalytics.com) [see 19, 20]. Very recent studies used Overton data [e.g., 11]. In the following, we summarize some of these policy impact studies chronologically. One of the first studies in this new altmetrics area was published by Bornmann, Haunschild [21] using an extensive publication set of climate change papers. The authors were interested in the question of how intensively policy documents have cited science publications. Although climate change is an ongoing policy topic worldwide, they found that only 1.2% out of 191,276 papers on cli- mate change in the dataset have at least one policy citation (using data from Altmetric). The results of Bornmann, Haunschild [21] revealed that review papers were more frequently cited in policy documents than articles. In order to investigate whether the percentage of 1.2% can be thought of as high or low, two of the authors investigated the percentage of papers indexed in the WoS that are mentioned in policy-related documents [22]. They found that less than 0.5% are mentioned at least once. Thus, the results show that although only 1.2% of climate

change papers were relevant for policy documents, this percentage is substantially higher than the percentage among all papers from the database. Vilkins and Grant [18] did not use data from Altmetric or PlumX for their empirical study, but used publications from policy-focused Australian Government departments. The authors were interested in the research and reference practices of Australian policymakers. The study is based on 4,649 cited references in 80 government publications from eight departments. They found that mostly peer-reviewed journal articles, federal government reports, and Aus- tralian business information have been cited. The study also revealed “a possible increased chance for academic research to be cited if it was open access. Despite criticisms of citation analysis, at least in the field of research utilisation we cannot solely rely on interview or survey data, as cited evidence use differs from reported evidence use” [18]. Tattersall and Carroll [23] used Altmetric policy documents data to investigate policy impact of papers published by authors at the University of Sheffield. They found that 0.65% of the papers were cited by at least one policy document. This percentage is slightly higher than that mentioned by Haunschild and Bornmann [22] for the WoS database. The field-specific policy-impact analysis revealed that “the research topics with the greatest policy impact are medicine, dentistry, and health, followed by social science and pure science” [23]. In a more

recent study, Yang, Huang [9] used the Chinese database iPolicy that includes policy docu- ments issued by the Chinese government since 1949. The authors used the data to construct networks of policy-making ministries and government departments. They were interested in identifying core policymakers in China and possible changes of their positions in the networks. Yang, Huang [9] present 15 ministries in China with the highest eigenvector centrality as core government ministries in the policy networks. Fang, Costas [24] focused on hot research topics reflected by citations in policy documents (using Altmetric.com data). The study is based on more than 10 million WoS papers published in various disciplines. The authors identified the hot topics in various broad disciplines. For example, they found that infectious diseases were typically of concern to policy-makers, but also topics that focus on industry and finance as well as child and education. In addition, “potential health-threatening environment problems (e.g., ‘ambient air pollution’, ‘environmental tobacco smoke’, ‘climate change’, etc.) drew high levels of attention from policy-makers too” [24]. Hicks and Isett [25] published a case study that investigated the policy impact of papers published in the area of quantitative studies of science. The authors speculated that many PLOS ONE How relevant is climate change research for climate change policy? PLOS ONE | https://doi.org/10.1371/journal.pone.0274693 September 22, 2022 4 / 24