Dataset search in biodiversity research: Do metadata in data repositories reflect scholarly information needs?

We are aware that this result is not complete and leaves room for improvement. Some cate- gory names were misleading and confused the annotators. That is reflected in fair and bad agree- ment for some categories such as QUALITY (data parameters measured) or DATA TYPE (nature or genre of the primary data). Here, it should be discussed in the research community how they could be further considered in search, e.g., re-naming or merging of categories. Since the backgrounds of the annotators were quite diverse and no training took place, we did not expect completeness and perfect agreement. We wanted to get a real and unbiased first picture of biodiversity scholars’ comprehension when looking for scientific data. In biology and biodiver- sity research, scholars use a specialized language with diverse content and imprecise and incon- sistent naming [31, 32]. Hence, labeling and extracting biological entities remain a challenge. Concerning the shortened methodology for the evaluation, our assumptions have been con- firmed. All nine domain experts completed the classification task. It saved a lot time that only a few people did the identification of artifacts to be labeled and that domain experts assigned categories, only. On average, domain experts spent between two and three hours for labeling 169 questions. We conclude that our shortened annotation approach is fine for opening up new domains and getting insights in what scholars are interested in. If the aim is to achieve higher agreement per annotation, we recommend training sessions and trial rounds. However,

such methods may result in biased feedback. For further reuse of the annotated question corpus, our analysis script also produces an XML file with all questions and annotations above a certain agreement threshold that can be set as a parameter. By default, all annotations per question with an agreement above 0.6 will be returned. B—Metadata standards in the life sciences In this section, we describe a selection of existing metadata standards and investigate whether their elements reflect the identified information categories. Fig 4. Frequency of category mentions and inter-rater agreement with QUALITY correction. https://doi.org/10.1371/journal.pone.0246099.g004 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 15 / 36