Dataset search in biodiversity research: Do metadata in data repositories reflect scholarly information needs?

entity types. Manually generated annotations are helpful in understanding what information users are interested in and support the development of tools and services to either automati- cally extract these interests from text (text mining), to retrieve relevant data (information retrieval) or to provide an exact answer for that information need (question answering). In the Life Sciences, question corpora for text retrieval have been mainly established in the medical and bio-medical domains. Genuine user requests have been collected in surveys for the Genomics Track at TREC conferences [29] and in email requests (67%) and search query logs (33%) for the Consumer Health Corpus [9]. In contrast, the question corpora for the BioASQ challenge [10], an annual challenge for researchers working on text mining, machine learning, information retrieval and question answering, was created and annotated by a team of 10 experts [30] with concrete requirements. Each expert was asked to formulate 50 questions in English that reflect “real-life information needs”. However, the type of questions to be for- mulated was restricted, e.g., the experts were asked to provide questions of certain types typi- cally considered in question answering systems (yes/no, factoid, etc.). These restrictions are justified to a certain degree since they affect the applicability of the resulting corpus for evalua- tion purposes of question answering approaches. However, they have an impact on which questions are formulated and how. A question corpus dedicated for dataset search retrieval is

included in the benchmark developed for the 2016 bioCADDIE Dataset Retrieval Challenge [25]. It includes 137 questions, 794,992 datasets gathered from different data portals in XML structure, and relevance judgments for 15 questions. Similar to the BioASQ challenge, domain experts had to consider guidelines on how to create questions. Based on templates, the ques- tion constructors formulated questions using the most desired entity types, namely data type, disease type, biological process, and organism. At present, to the best of our knowledge, there is neither a public log analysis nor a question corpus available for biodiversity research. In order to understand genuine user interests and to improve current dataset retrieval systems, unfiltered information needs are crucial. Therefore, collecting current search interests from scholars is the first step in our top-down approach pre- sented in Section “A—Information Needs in the Biodiversity Domain”. Dataset search Dataset search is mainly based on metadata describing heterogeneous primary data. Terminol- ogies used in metadata are influenced by scholars’ research fields. As data in the Life Sciences are difficult to classify and categorize for humans [31, 32], it is an even larger challenge to develop a system that can interpret user queries correctly and return relevant datasets. The main aim of a retrieval process is to return a ranked list of datasets that match a user’s query. Based on the underlying Retrieval Model, different ranking functions have been devel-

oped to produce a score for the datasets with respect to the query. Top-scored datasets are returned first. Classical retrieval models include the Boolean Model [12], where datasets are only returned that exactly match a query, and the Vector Space Model [12], often used in com- bination with the Boolean Model. In the Vector Space Model, the content of datasets is repre- sented by vectors that consist of term weights. The similarity of datasets and queries is determined by computing the distance between the vectors. All these retrieval methods have in common that they only return datasets that exactly match a user’s query. A study by the RDA Data Discovery Paradigm Interest Group points out [4] that most data repositories utilize either Apache Solr (http://lucene.apache.org/solr/) or elasticsearch (https://www.elastic.co/products/elasticsearch). The default retrieval method in these search engines is either one of the mentioned keyword-based retrieval models or a combination of both. Apache Solr and elasticsearch provide fuzzy-search PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 6 / 36