Dataset search in biodiversity research: Do metadata in data repositories reflect scholarly information needs?

We are aware that this result is not complete and leaves room for improvement. Some cate- gory names were misleading and confused the annotators. That is reflected in fair and bad agree- ment for some categories such as QUALITY (data parameters measured) or DATA TYPE (nature or genre of the primary data). Here, it should be discussed in the research community how they could be further considered in search, e.g., re-naming or merging of categories. Since the backgrounds of the annotators were quite diverse and no training took place, we did not expect completeness and perfect agreement. We wanted to get a real and unbiased first picture of biodiversity scholars’ comprehension when looking for scientific data. In biology and biodiver- sity research, scholars use a specialized language with diverse content and imprecise and incon- sistent naming [31, 32]. Hence, labeling and extracting biological entities remain a challenge. Concerning the shortened methodology for the evaluation, our assumptions have been con- firmed. All nine domain experts completed the classification task. It saved a lot time that only a few people did the identification of artifacts to be labeled and that domain experts assigned categories, only. On average, domain experts spent between two and three hours for labeling 169 questions. We conclude that our shortened annotation approach is fine for opening up new domains and getting insights in what scholars are interested in. If the aim is to achieve higher agreement per annotation, we recommend training sessions and trial rounds. However,

such methods may result in biased feedback. For further reuse of the annotated question corpus, our analysis script also produces an XML file with all questions and annotations above a certain agreement threshold that can be set as a parameter. By default, all annotations per question with an agreement above 0.6 will be returned. B—Metadata standards in the life sciences In this section, we describe a selection of existing metadata standards and investigate whether their elements reflect the identified information categories. Fig 4. Frequency of category mentions and inter-rater agreement with QUALITY correction. https://doi.org/10.1371/journal.pone.0246099.g004 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 15 / 36

Methodology Metadata can be considered as additional information about primary data such as experi- ments, tabular data, images, sound and acoustic files to make scientific data understandable and machine-readable. In order to enable machine processing, metadata in the Life Sciences are mostly stored in structured formats such as XML or JSON. A metadata schema [56] for- mally describes the syntax and structure of metadata, e.g., which elements and attributes can be used and which elements are mandatory and/or repeatable. If a schema has become estab- lished in a research community, it is usually adopted as a metadata standard. Another oppor- tunity to become a metadata standard is the formal adoption by a standards’ organization such as the International Organization for Standardization, https://www.iso.org. In this work, we use the term ‘metadata standard’ in a liberal way and include both schemata being adapted by the community and by an organization. We use the term ‘metadata field’ when discussing indi- vidual elements of a concrete metadata file. The term ‘metadata structure’ comprises all used fields in a metadata file. As the outcome of the question analysis reveals (Section “A—Information Needs in the Bio- diversity Domain”), biodiversity research is a broad field spanning a variety of disciplines. Therefore, in the following analysis, we selected metadata standards from two sources (re3data [57] and RDA Metadata Standards Catalog (version 2) [58]) with a broader perspective. In par-

ticular, we wanted to focus on metadata standards being used in current data portals and archives. In re3data, we filtered for “Life Sciences” (https://www.re3data.org/search?query= &subjects%5B%5D=2%20Life%20Sciences). Under the facet ‘Metadata standards’, a list of 24 standards appeared that we used for our further analysis. From RDA Metadata Standards Cat- alog, we selected all top-level standards (https://rdamsc.bath.ac.uk/scheme-index) labeled with “Science” (https://rdamsc.bath.ac.uk/subject/Science) resulting in a list of 30 standards. We merged both lists and cleaned the final outcome according to the following criteria: The cate- gories Other and Repository-Developed Metadata Schema have been omitted. The MIBBI stan- dard is outdated and has been integrated into ISA-Tab, so we left it out, too. The same applies to the Observ-OM and CIM standard. The information on the website is deprecated and not fully available (e.g., dead links). We also omitted the “Protocol Data Element Definitions” (data elements that are required for data archival for clinical trials), all astronomy and astro- physics related standards and standards for social and behavioral studies, as these fields are out of our scope. Table 2 presents the final list of 21 metadata standards used for the following analysis. They are ranked by the number of data repositories supporting them (obtained from re3data). We analyzed whether the standard supports semantic web formats, e.g., RDF or OWL. According to the FAIR principles [11], community standards, semantic formats, and

ontologies ensure interoperability and data reuse. The last column provides some examples of data repositories supporting the standard. Further, supplementary material on the selec- tion process can be found in our GitHub repository (https://github.com/fusion-jena/ QuestionsMetadataBiodiv/tree/master/metadataStandards). The standard supported by most repositories is Dublin Core, a general metadata standard based on 15 fields, such as contributor, coverage, creator, date, description, format, and identi- fier. In addition, data repositories utilize further domain-specific standards with richer vocab- ulary and structure such as ISO19115 for geospatial data or EML for ecological data. The RDF Data Cube Vocabulary is not used by any of the data centers. We suppose, the abbreviation RDF DC might lead to a misunderstanding (Dublin Core instead of RDF Data Cube). All stan- dards provide elements that can be described along the questions: Who? What? Where? When? Why? and How?. In particular, contact person, collection or publication date and PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 16 / 36

location are considered with one or several metadata fields in all standards. In order to describe the main scope of the primary data, all standards offer numerous metadata fields but differ in their granularity. While simple ones such as Dublin Core only offer fields such as title, description, format, and type, standards with more elements such as EML or ABCD even offer fields for scientific names, methods and data attributes measured. EML even allows scholars to define the purpose of the study making it the only standard that supports the Why question. Data reuse and citation also play an important role. As it is demanded by the Joint Declaration of Data Citation Principles [59] and practical guidelines for data repositories [60], all standards Table 2. Metadata standards in the (life) sciences obtained from re3data [57] and RDA metadata standards catalog [58]. The number in brackets denotes the number of repositories supporting the standard (provided in re3data). Standard Name URL Domain Semantic Format Examples Dublin Core (205) http://dublincore.org/ general research data Yes (RDF) Pangaea, Dryad, GBIF, Zenodo, Figshare DataCite (111) https://schema.datacite.org/meta/kernel-4.1/ general research data No Pangaea, Zenodo, Figshare, Radar ISO19115 (47) ftp://ftp.ncddc.noaa.gov/pub/Metadata/Online_ ISO_Training/Intro_to_ISO/workbooks/MD_ Metadata.pdf geospatial data No Pangaea, NSF Arctic Data Center, coastMap FDGC/ CSDGM (42) https://www.fgdc.gov/metadata/csdgm-standard geographic information No Dataverse, NSF Arctic Data Center

Darwin Core (28) https://dwc.tdwg.org/ biodiversity data Yes (RDF) GFBio, GBIF, VerNET, Atlas of Living Australia, WORMS EML (26) https://knb.ecoinformatics.org/external// emlparser/docs/eml-2.1.1/index.html ecological data No GBIF, GFBio, SNSB, Senckenberg, WORMS, NSF Arctic Data Center RDF Data Cube (20) https://www.w3.org/TR/vocab-data-cube/ statistical data Yes Dryad (only RDF with DublinCore) ISA − Tab (13) https://isa-specs.readthedocs.io/en/latest/index. html biological experiments Yes Data Inra, GigaDB ABCD (11) https://github.com/tdwg/abcd biological collection data Yes (ABCD 3.0 in RDF) GBIF, BioCase Network OAI − ORE (11) https://www.openarchives.org/ore/ general research data Yes (RDF) Environmental Data Initiative Repository, Dryad DCAT(9) https://www.w3.org/TR/vocab-dcat data catalogs, data sets Yes Data.gov.au, European Data Portal DIF(9) https://gcmd.nasa.gov/DocumentBuilder/ defaultDif10/guide/index.html geospatial metadata No Pangaea, Australian Antarctic Data Center, Marine Environmental Data Section CF(7) http://cfconventions.org/ climate and forecast No WORMS, NSF Arctic Data Center, coastMap SDMX (4) https://sdmx.org/ statistical data No UN Data Catalog CSMD − CCLRC (1) http://icatproject-contrib.github.io/CSMD/ scientific activities, e.g., experiments, observations, simulations Yes (OWL) Monash University Research Repository Genome Metadata (1) https://docs.patricbrc.org/user_guides/ organisms_taxon/genome_metadata.html genome data No PATRIC OME − XML https://docs.openmicroscopy.org/ome-model/6.2. 0/

biological imaging Yes (OWL) CELL Image Library, JCB Dataviewer PDBx/mmCIF http://mmcif.wwpdb.org/dictionaries/mmcif_ pdbx_v40.dic/Index/index.html 3D structures of proteins, nucleic acids, and complex assemblies No wwPDB (Worldwide Protein Data Bank) UKEOF http://www.ukeof.org.uk/schema/ environmental monitoring No UKEOF catalogue CEDAR https://more.metadatacenter.org/tools-training/ outreach/cedar-template-model general but focus on scientific experiments Yes (RDF) US National Cancer Institute’s caDSR Observation and Measurement (OM) https://www.ogc.org/standards/om observational data Yes (OWL-Lite) INSPIRE (Infrastructure for Spatial Information in Europe) https://doi.org/10.1371/journal.pone.0246099.t002 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 17 / 36

provide several elements for digital identifiers, license information and citation. In addition, some standards provide elements for data quality checks. For instance, ISO19115 offers a con- tainer for data quality including lineage information and EML supports quality checks with the qualityControl element. Increasingly, semantic formats are supported. While some standards already provide or are currently developing fully semantic standards (ABCD, Dar- win Core, OAI-ORE, OME-XML and CEDAR), others provide elements to link taxonomies (Genome Metadata, PDBx/mmCIF) or started integrating controlled vocabularies (ISA-Tab, UKEOF, CEDAR). Results In our second analysis, we compared the information categories with elements of the metadata standards to figure out, if search interests can be explicitly described with metadata elements. Our results are presented in Table 3. The detailed matching between categories and metadata fields is available as csv file in our GitHub repository. For the sake of completeness, we explored all 13 categories from the previous analysis but marked the ones with an asterisk that had a fair agreement (< 0.4). The categories are sorted by frequency from left to right. The red color denotes that no element is available in the standard to express the category, orange indi- cates that only a general field could be used to describe the category and a yellow cell implies that one or more elements are available in the standard for this search interest.

Table 3. Comparison of metadata standards and information categories. The categories are sorted by the frequency of their occurrence determined in the previous question analysis, the asterisk denotes the categories with an agreement less than 0.4. Environment Quality� Material Organism Process Location Data Type� Method� Anatomy� Human Intervention� Event� Time Person DublinCore DataCite ISO19115 FDGC/CSDGM EML DarwinCore RDFDataCube ISA − Tab DIF CF ABCD DCAT OAI − ORE SDMX Genome Metadata OME-XML PDB/mmCIF UKEOF CSMD-CCLRC CEDAR OM Table key Red: Not provided, Orange: Unspecific (general element), Yellow: Available (one or more elements) https://doi.org/10.1371/journal.pone.0246099.t003 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 18 / 36

There is no schema that covers all categories. Since the interests are obtained from scholars with various and heterogeneous research backgrounds, this was to be expected. Apart from HUMAN INTERVENTION, all categories are covered by different metadata standards. In particular, ISA-Tab covers most of the search interests of biodiversity researchers explicitly. The description of QUALITY information (e.g., data parameters measured) are covered by a variety of standards, e.g., ABCD, Genome Metadata, SDMX. The same applies for information on the DATA TYPE and geographic information (LOCATION). For the latter, the granularity differs between the individual standards. While standards such as ISO19115, ABCD, Genome Metadata, Data Cite offering several metadata fields for the description of geographic informa- tion, other standards only provide one metadata element, e.g., CSMD, EML, Dublin Core. Metadata elements for environmental information are provided by EML (studyAreaDe- scription), UKEOF (environmentalDomain) or CSDM (Facility). Observed or involved organisms can be described with Darwin Core, ABCD or EML but also with standards focusing on molecular data (ISA-Tab, Genome Metadata and PDB/mmCIF). However, important search preferences such as materials (including chemicals) are only explicitly supported by ISA-Tab, PDB/mmCIF, CSMD and OM. While numerous standards already support the description of research methods and instruments (METHOD), only ISA- Tab offers a metadata field (Protocol REF) to describe biological or chemical processes

(PROCESS) a specimen or an experiment is involved. Even if some categories are not fre- quently mentioned (Subsection “Results and Metrics”), such as HUMAN INTERVENTION and EVENT, they should not be neglected. Extreme weather events or incidents, e.g., earth- quakes or oils spills, might be of research interest and therefore, metadata elements should be provided to define and describe them. Widely used general standards such as Dublin Core or DataCite offer at least a general field (dc:subject, subject) that could be used to describe the identified search categories. In Dublin Core, at least one metadata field each is provided to describe geographic information, e.g., where the data have been collected (LOCATION), the type of the data (DATA TYPE), the creator and contributor (PERSON & ORGANIZATION) and when it was collected or pub- lished (TIME). However, one field is often not enough to distinguish if the provided field is for instance a collection date or publication date, or if the creator of the dataset is also the person that collected the data. In contrast, DataCite provides individual fields for publication year and the date field can be used with dateType=“Collected” to specify a collection date. The metadata field contributor can also be extended with a type to indicate whether the con- tact details belong to the data collector or the project leader. Bounding box elements are also provided to enter geographic coordinates (LOCATION). A complete generic approach is CEDAR offering generic elements to set up metadata templates in various formats (e.g., JSON,

RDF) and options for integrating controlled vocabularies or schema.org entities. Summary. Generic metadata standards such as Dublin Core or DataCite do not describe primary data sufficiently for dataset search. They cover main scholarly search interests only if general fields are used and proper keywords (in best-case using controlled vocabularies) are contained. A variety of discipline-specific standards are available that go beyond data citation aspects and cover most of the important topics for search, e.g., ISA − Tab (biomedical experi- ments), Darwin Core, ABCD (species), EML (ecology), ISO19115 (geospatial data), CSMD, PDB/mmCIF and ISA − Tab (materials). However, PROCESS is currently only explicitly cov- ered by ISA-Tab. Here, the respective research communities should consider to add elements to the standards that allow an explicit description or at least to provide general fields that allow an extension. The question that still remains to be answered is what metadata standards are actually used by data repositories. PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 19 / 36

C—Metadata usage in selected data repositories In the following analysis, we examine how well existing metadata cover scholarly information needs in biodiversity research. Therefore, we analyzed metadata from selected data repositories along the following criteria: • Statistics: At first, we inspected what metadata standards are offered in the repositories’ metadata API. For the further processing, we only selected standards that are relevant for biodiversity research. • Timelines: Timelines indicate how many datasets per year are published in a data repository and when particular standards have been introduced. • Metadata Field Usage: In order to get an overview of which metadata elements are used we determined how many fields of a standard are filled. • Category-Field-Match: Focusing on the standard that best covers the identified categories, we explored what fields reflect information needs for biodiversity research and how often they are filled. • Content Analysis: Counting alone does not provide further insights on the actual content in the metadata. Therefore, we analyzed free text metadata fields such as title, description and further general fields, and we inspected whether they contain useful information for search. Keywords in general fields were extracted and sorted by frequency for all parsed files and from all data repositories. A deeper content analysis of the descriptive metadata fields was limited to 10,000 files per data repository due to performance limitations of the used text mining tools.

We first introduce the methodology comprising data collection, data processing and the used strategy and tools for the content analysis. Afterwards, we present the results along the introduced criteria. Methodology Scholarly publishers increasingly demand scientific data to be submitted along with publica- tions. Since publishers usually do not host the data on their own, they ask scholars to publish data with a repository for their research domain. According to Nature’s list of recommended data repositories [16], we selected five archives for our further analysis: three generalist ones (Dryad, Figshare and Zenodo) and two domain-specific ones (PANGAEA—environmental data, GBIF—taxonomic data). In the biodiversity projects we are involved in, scholars also often mention these repositories as the ones they mainly use. Data collection. We parsed all available metadata from Figshare, Dryad, GBIF, PANGAEA and Zenodo in May 2019 via their respective OAI-PMH (https://www.openarchives.org/pmh/) interfaces. The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a protocol primarily developed for providing and consuming metadata. GBIF only offers the metadata of their datasets in the OAI-PMH interface. The individual occurrence records, which belong to a dataset and which are provided in metadata structures based on the Darwin Core metadata standard [61], are not present in the OAI-PMH interface. Hence, we only ana- lyzed the metadata of the datasets. Our script parses the metadata structure of all public records per metadata standard for

each of the selected data repositories (Table 4). Apart from the metadata standards introduced in the previous section, a few more standards appear in this list. OAI-DC is an abbreviation for Dublin Core, a mandatory standard in OAI-PMH interfaces. QCD means qualified Dublin PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 20 / 36

Core and denotes an extended Dublin Core extending or refining the 15 core elements. We also considered Pan-MD, a metadata schema developed by PANGAEA. It extends Dublin Core with more fine-grained geographic information such as bounding boxes or adds information on data collection. The latter can range from projects, parameters, methods, and sensors to taxonomy or habitats. MARC21, MARCXML and METS are metadata standards that are mainly used for biblio- graphic data in digital libraries. Hence, we left them out of our further explorations. We also did not consider the Common European Research Information Format (CERIF) and OAI-ORE as they are not focused on describing primary data but research entities and their relationships and grouping web resources, respectively. However, we decided to permit all available reposi- tory-developed schemata for the Life Sciences, such as Pan-MD, in order to get an impression how repositories extend metadata descriptions. RDF is not a standard but a semantic format. We decided not to omit it as it indicates the usage of semantic formats at data repositories. The metadata fields used in RDF format are mainly based on Dublin Core. Data processing. Per metadata file we inspected which elements of the metadata stan- dards are used, and we saved their presence (1) or non-presence (0). The result is a csv file per metadata standard containing dataset IDs and metadata elements used. All generated files are stored in separate folders per repository and metadata standard. Each request to a repository

returns an XML body that includes several metadata files as records. Each record is separated in two sections, a header and a metadata section. The header section comprises general infor- mation such as a unique identifier of the item and a date stamp. The metadata section contains the metadata structure, e.g., the name of the contributors, abstract and publication year. Unused metadata fields are not included in the response. We saved a boolean value encoding whether a metadata field was used or not. The source code and documentation on how to use it is available in our GitHub repository. For our further consideration, we wanted to obtain a publication date of each downloaded dataset to inspect how many datasets have been published over the years in which standard per data repository. Unfortunately, a publication date is not provided in all metadata schemata. Therefore, we looked up each date related field in the schema and used the one that is (based on the description) the closest to a publication date. Table 5 depicts all date stamps utilized and their descriptions. If the respective date stamp was not found in a dataset or was empty, we left the dataset out in the following analysis. Data collection for content analysis. General, descriptive metadata fields such as ‘title’, ‘description’ or ‘abstract’, and ‘subject’ might contain relevant data that are interesting for information seekers. Using conventional retrieval techniques, this data is only accessible in a full text search and if the entered query terms exactly match a term in the dataset. These

Table 4. Metadata schemes and formats offered by selected data repositories in their OAI-PMH interfaces. Dryad GBIF PANGAEA Zenodo Figshare METS EML DATACITE3 DATACITE CERIF OAI-DC OAI-DC DIF DATACITE3 METS OAI-ORE ISO19139 DATACITE4 OAI-DATACITE RDF ISO19139.IODP MARCXML OAI-DC OAI-DC MARC21 QDC PAN-MD OAI-DATACITE RDF OAI-DATACITE3 OAI-DC https://doi.org/10.1371/journal.pone.0246099.t004 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 21 / 36

metadata fields potentially contain the most relevant search terms and are also candidates to be enriched with text mining tools. Hence, we aim to explore what information is currently available in general, descriptive metadata fields. We downloaded descriptive metadata fields, namely, dc:title, dc:description and dc:subject in OAI-DC format from all repositories in October and November 2019. Parallel to the download, we collected the key- words used in the subject field and counted their presence in a separate csv file. In order to further inspect the content with Natural Language Processing (NLP) [24] tools, we selected a subset of representative datasets. We limited the amount to 10,000 datasets per repository as the processing of textual resources is time-consuming and resource-intensive. A variety of applications have been developed to determine Named Entities (NE) such as geo- graphic locations, persons and dates. Thessen et. al [31] explored the suitability of existing NLP applications for biodiversity research. Their outcome reveals that current text mining sys- tems, which were mainly developed for the biomedical domain, are able to discover biological entities such as species, genes, proteins and enzymes. We concentrated on the extraction of entity types that (a) correspond to the identified search interests (Section “A—Information Needs in the Biodiversity Domain”) and for which (b) text mining pipelines are available. We used the text mining framework GATE [62] and its ANNIE pipeline [63] as well as the Orga-

nismTagger [64] to extract geographic locations, persons, organizations and organisms. In our previous research, we developed the BiodivTagger [65], a text mining pipeline extracting envi- ronmental terms, materials, processes and data parameters. Results In this subsection, we present the results of the harvested and processed metadata. We ana- lyzed the data along the introduced criteria, beginning with statistics and timelines, a study on the use of metadata fields and a deeper content analysis of metadata fields. Statistics. The overall statistics of the analyzed data repositories are presented in Table 6. Most repositories support general standards for metadata creation; only PANGAEA and GBIF utilize metadata based on discipline-specific metadata standards. Dryad and Figshare already provide metadata in semantic formats such as RDF. In addition, Figshare offers Qualified Dub- lin Core (QDC), an extended Dublin Core that allows the description of relations to other data sources. We checked the fields for a publication date in a valid format. We could not use all harvested datasets as for some metadata files publication dates were not available. Dryad had a Table 5. The date stamps used for each metadata standard and their descriptions obtained from the standard’s website. Format Element URL Description OAI-DC/METS/QDC RDF(Dryad) dc:date http://www.dublincore.org/specifications/dublin-core/ dces/ “A point or period of time associated with an event in the lifecycle of the resource.” EML pubDate https://knb.ecoinformatics.org/external//emlparser/docs/

eml-2.1.1/eml-resource.html#pubDate “The ‘pubDate’ field represents the date that the resource was published.” DATACITE3/4 OAI-DATACITE/3 publicationYear https://support.datacite.org/docs/schema-40 “The year when the data was or will be made publicly available.” DIF DIF_Creation- _Date https://gcmd.gsfc.nasa.gov/DocumentBuilder/ defaultDif10/guide/metadata_dates.html “refers to the date the data was created” ISO19139/ISO19139.iodp gco:DateTime https://geo-ide.noaa.gov/wiki/index.php?title=ISO_Dates (CI-DataTypeCode=publication), publication Date PAN-MD md:dateTime http://ws.pangaea.de/schemas/pangaea/MetaData.xsd publication date (contact to data repository) RDF(Figshare) vivo: datePublished https://codemeta.github.io/terms/ “Date of first broadcast/publication” https://doi.org/10.1371/journal.pone.0246099.t005 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 22 / 36

large number of datasets with a status “Item is not available”, which we left out, too. The num- ber in brackets denotes the amount of datasets we used for the following considerations. Timelines. Based on the given publication dates, we computed timelines (Fig 5) for the introduction of the various standards over time per data repository. The code and all charts are available as supplementary material in our GitHub repository. As Dryad provides several dc:date elements in the metadata, we used the first available date entry as publication date for the timeline chart. Per repository, the timelines for the different metadata formats are almost identical. Obvi- ously, when introducing a new metadata format, publication dates were adopted from existing metadata formats. Only Figshare uses new date stamps when a new metadata format is pro- vided. For instance, Figshare’s timeline shows that QDC and RDF were launched in 2015. The result for RDF was too large to process it together with the other metadata formats. Hence, we produced the timeline for RDF separately. The timelines across all repositories reveal a steadily increasing number of datasets being published at GBIF, Dryad, Zenodo and Figshare. For PAN- GAEA, the timeline points to a constant number of published datasets of around 10,000 data- sets a year apart from an initial release phase between 2003 and 2007. Metadata field usage. Fig 6 presents how many metadata elements of the best matching standard were filled. The individual results per data archive are available in our repository as

supplementary material (https://github.com/fusion-jena/QuestionsMetadataBiodiv/tree/ master/data_repositories/charts). Dryad used 9 out of 15 available metadata fields from OAI-DC very often (> 80%) including important fields such as dc:title, dc:descrip- tion and dc:subject. dc:publisher and dc:contributor were provided in less that 20%. For GBIF, the EML standard does not provide a fixed number of core elements. Hence, we analyzed the 129 available fields. Most of them (89 elements) were not filled, e.g., fields describing taxonomic information. Data about author, title and description were pro- vided in more than 80%. The general field eml:keyword was used in around 20%. Out of 124 used fields in PANGAEA’s Pan-MD format, 43 fields were filled in more than 80% of the harvested metadata files including information on the author, project name, coordinates, data parameters and used devices. Fields that were less filled are supplementary fields, for instance for citation, e.g., volume, pages. For Zenodo, all required fields in DataCite (identifier, creator, title, publisher, publication year) were always filled. In Table 6. Total number of datasets parsed per data repository and metadata standards and schemata. The numbers in brackets denote the number of datasets used for the analysis. All datasets were harvested and parsed in May 2019. Metadata Standard Dryad PANGAEA GBIF Zenodo Figshare OAI-DC 186951 (142329) 383899 (383899) 44718 (42444) 255000 (255000) 3128798 (3128798) QDC 1718059 (1718059) RDF 186955 (142989) 3157347 (3157347) DATACITE 1268155 (1268155)

DATACITE3 383906 (383906) 1268232 (1268232) OAI-DATACITE 1266522 (1266522) 3134958 (3134958) OAI-DATACITE3 1268679 (1268679) DATACITE4 1268262 (1268262) EML 44718 (42444) DIF 383899 (383899) ISO19139 383899 (383899) ISO19139.iodp 383899 (383899) PAN-MD 383899 (383899) https://doi.org/10.1371/journal.pone.0246099.t006 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 23 / 36

Fig 5. Timelines for all repositories presenting the number of datasets per metadata standard and schema offered. For several repositories, the timelines for the different metadata standards and schemata are almost identical and overlap. Obviously, when introducing a new metadata standard or schema, publication dates were adopted from existing metadata structures. Figshare’s timeline for RDF was computed separately as the data are too large to process it together with the other metadata files. https://doi.org/10.1371/journal.pone.0246099.g005 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 24 / 36

addition, title, rights and descriptions as well as resource type were also provided in more than 99% of the analyzed metadata files. However, in only 45% of the metadata files, keywords (subject) were present. Figshare used only 12 out of 17 available fields of QDC, but these fields were always filled. 5 out of 17 fields of QDC were not used at all. Fig 6. Metadata field usage in all data repositories evaluated. The graphics display the percentage of metadata fields used per data repository and its best matching standard with respect to the information categories. https://doi.org/10.1371/journal.pone.0246099.g006 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 25 / 36

Category—Field—Match. Per data repository and metadata standard, we computed charts that visualize which field was filled at what percentage rate and if they correspond to the categories introduced in Section “A—Information Needs in the Biodiversity Domain”. Table 7 presents a summary of all data repositories and their best matching standard. The indi- vidual results per repository and the concrete field-to-category mapping are available in our repository. Temporal expressions (TIME) and information about author and/or creator (PERSON) were mostly provided in all repositories. Apart from PANGAEA, repositories mainly provided the publication date and only partially added information about when the data was collected. Information about data type and formats was also contained in all metadata files apart from GBIF. The identified search categories were partially covered by two repositories. GBIF with EML reflects most of the categories, but fields that correspond to ENVIRONMENT, ORGAN- ISM, DATA TYPE and METHOD were rarely filled. Metadata files in PANGAEA0 s reposi- tory-developed standard Pan-MD always contained information on data parameters (QUALITY) and geographic locations. In most cases, research methods and devices used were also given. Dryad provided geographic information (LOCATION) in its dc:coverage field in at least 60%. Content analysis. Table 8 presents the five most common keywords in the metadata field dc:subject for all repositories sorted by their frequencies. The full keyword lists are avail- able in our repository.

For GBIF datasets, in 81% an empty dc:subject field was returned. Zenodo’s metadata provided keywords in 52% of the inspected cases. There are inconsistencies in how terms are entered; for example, none of the repositories seem to consider upper and lower cases. For sev- eral terms, different spellings resulted in separate entries. Numerous keywords in PANGAEA Table 7. Comparison of data repositories and their best matching standard with the information categories. The categories are sorted by the frequency of their occur- rence determined in the question analysis. The asterisk denotes the categories with an agreement less than 0.4. Environment Quality� Material Organism Process Location Data Type� Method� Anatomy� Human Intervention� Event� Time Person GBIF (EML) (3%) (11%) (35%) (8%) (18%) (publication date—100%, collection data—10%) (>90%) Dryad (OAI − DC) (60%) (publication date) (80%) PANGAEA (Pan − MD) (>90%) (100%) (100%) (Devices used— 90%, research methods —65%) (publication Date—100%, collection date—80%) (100%) Zenodo (OAI − Datacite) (100%) (publication Date) (100%) Figshare (QDC) (100%) (publication Date) (100%) Table key Orange: Unspecific (generic element), Yellow: Available (one or more elements) Amount in brackets denotes the percentage the element is filled. https://doi.org/10.1371/journal.pone.0246099.t007 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 26 / 36

and Dryad reveal that both repositories host marine data. PANGAEA’s list mainly contains data parameters measured and used devices. In contrast, Dryad’s list indicates that terrestrial data are also provided. For instance, the lower ranked terms contain entries such as Insects (1296) (insects (180)) or pollination (471) (Pollination (170)). Geographic information, e.g., California (9817), also occurred in Dryad’s dc:subject field. Zenodo’s and Figshare’s key- word lists contain numerous terms related to collection data. We checked the term ‘Biodiver- sity’ in both repositories in their search interfaces on their websites. It turned out that the Meise Botanic Garden (https://www.plantentuinmeise.be) provided large collection data in Zenodo. Hence, each occurrence record counted as a search hit and got the label ‘Biodiversity’. We also discovered that Figshare harvests Zenodo data which also resulted in high numbers for Figshare and the keyword ‘Biodiversity’ (219022). In a second analysis, we investigated entity types that occur in descriptive metadata fields. As the processing of textual resources with NLP tools is time-consuming and resource-inten- sive, we selected a subset of datasets. We limited the amount to 10,000 datasets per repository. Table 9 presents the filter strategies. For PANGAEA and GBIF, we randomly selected 10,000 datasets as they are domain-specific repositories for which all data are potentially relevant for biodiversity research. For Zenodo and Figshare we used the keyword ‘Biodiversity’. Due to the

large amount of collection data in Zenodo and Figshare with the keyword ‘Biodiversity’, we are aware that this filter strategy may have led to a larger number of collection datasets included in the 10,000 selected files. For Dryad, the filter contained a group of relevant keywords as a filter with only one keyword would not have returned at least 10,000 datasets. Per data repository, we processed the selected 10,000 files with open source taggers of the text mining framework GATE [62]. Named Entities such as PERSON, ORGANIZATION and LOCATION were obtained with the ANNIE pipeline [63], ORGANISMs were extracted with the OrganismTagger [64], and further biological entities (QUALITY, PROCESS, MATERIAL Table 8. Five most common keywords and their frequencies in the metadata field dc:subject. The last row denotes the amount of files with an empty dc:sub- ject field. PANGAEA GBIF Dryad Zenodo Figshare water (201102) Occurrence (6510), occurrence (46) Temperature (16652), temperature (15916) Taxonomy (459877), taxonomy (105) Medicine (1057684), medicine (240) DEPTH (198349), Depth (71916) Specimen (3046), specimen (22) Integrated Ocean Observing System (16373) Biodiversity (458336), biodiversity (8593) Biochemistry (1015906), biochemistry (92) Spectral irradiance (175373) Observation (2425), observation (24) IOOS (16373) Herbarium (270110), herbarium (91) Biological Sciences not elsewhere classified (983829) DATE/TIME (128917) Checklist (589), checklist (43) Oceanographic Sensor Data (15015) Terrestrial (269900), terrestrial (177) Chemical Sciences not elsewhere

classified (842865) Temperature (118522), temperature (50) Plantas (368), plantas (42) continental shelf (15015) Animalia (205242), animalia (261) Biotechnology (792223), biotechnology (23978) 0 38296 15436 705730 0 https://doi.org/10.1371/journal.pone.0246099.t008 Table 9. Filter strategies used per data repository to select 10,000 datasets. The number in brackets denotes the total number of available datasets (OAI-DC standard) at the time of download (October/November 2019). PANGAEA GBIF Dryad Zenodo Figshare 10000 randomly selected (388142) 10000 randomly selected GBIF (46954) 10000 randomly with keywords: biodiversity, climate change, ecology, insects, species richness, invasive species, herbivory, pollination, endangered species, ecosystem functioning, birds (149671) 10000 randomly with keyword: Biodiversity (1457958) 10000 randomly with keyword: Biodiversity (3592808) https://doi.org/10.1371/journal.pone.0246099.t009 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 27 / 36

the identification of biological entities. That could be an approach for generalist repositories to additionally enrich metadata. All scripts and the final results are available in our repository. Summary. In all repositories, we determined an increasing amount of datasets over the years. In some of them, the growth rate is exponential. This confirms the importance of data repositories and the increasing demand for data publication services. Our results reveal that generalist repositories tend to use only general standards such as Dublin Core and DataCite. Even if it is common understanding that generalist repositories only support general metadata standards, it would be a great advantage for data discoverability if generalist repositories offer several different metadata standards. This would additionally emphasize the message of gener- alist repositories to be able to handle any type of scientific data. Discipline-specific repositories, e.g., GBIF and PANGAEA are more likely to provide domain-related standards such as EML or Pan-MD. In GBIF’s case, we are aware that we could not provide a full picture as we did not analyze the occurrence records. Here, only a deeper analysis of the provided fields in the search index would deliver more answers. However, that would require technical staff support as the access to GBIF’s search indices is limited. Overall, the only metadata fields that matched researchers’ search interests closely were TIME and PERSON, which were present in almost all inspected repositories. However, impor-

tant categories such as information on environments, materials or species are only covered if Table 10. NLP analysis: Number of datasets with named entities (out of 10,000 processed files in a reduced OAI-DC structure) per repository. Each file contains a subset of the original metadata, namely, dc:title, dc:description, dc:subject and dc:date. PANGAEA GBIF Dryad Zenodo Figshare Location 9088 5718 3530 4644 9978 Person 3789 6687 3030 3201 1773 Organization 3307 1762 1486 1674 9542 Environment 2918 1533 5166 2050 15 Quality 7794 1830 6548 3183 29 Process 1756 1055 6743 1847 15 Material 8593 3358 3716 2859 644 Organism 3251 1217 3603 2891 8542 https://doi.org/10.1371/journal.pone.0246099.t010 PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 28 / 36

general fields or longer textual fields such as title and description contain useful keywords. Most repositories indeed enhance metadata with numerous keywords in general fields. They reflect scholarly information needs to some extent. The extracted keyword list emphasizes the heterogeneity of research data in biodiversity research and shows in which different granular- ity data are described. Keywords vary between broad subject categories and concrete measur- ing methods. In addition, different spellings amplify the problem of inconsistency. D—Discussion In this study, we explored what hampers dataset retrieval in biodiversity research. The follow- ing section discusses our findings, highlights topics that should be further discussed in the bio- diversity research community and lists remaining challenges for computer science. Scholarly search interests in biodiversity research We identified five information categories being important for search tasks in biodiversity research, namely, ENVIRONMENT, MATERIAL, ORGANISM, PROCESS and LOCATION. The categories QUALITY and DATA TYPE only got a fair agreement but are mentioned quite often in the search questions. Comparing our question analysis to the outcome of the content assessment analysis con- ducted in the GBIF community [28] in 2009, the authors’ assumption of this study that user interests change over time has been confirmed. Species are still an important category scholars are interested in, however, further important topics for the acquisition and description of eco-

system services are emerging. Our results confirm some of the defined dimensions in EASE [43], a metadata schema developed for search purposes. Common categories are ORGANISM, PROCESS, TIME and LOCATION (Space). Instead of the category QUALITY, data parame- ters in EASE are collected in sub-elements of the dimension METHOD. For some categories, our definitions go beyond the EASE dimensions. We found that MATERIALs in a broader sense are relevant to biodiversity researchers and not only chemicals. The same applies for environmental terms. They are in general important for dataset retrieval in biodiversity research and not just biomes. In addition to EASE we found that a separate category DATA TYPE should be considered as many researchers asked for specific result types of research methods. In summary, we conclude that the diversity of scholarly search interests in biodiversity research can indeed be structured and grouped into a limited amount of categories. Consider- ing these interests in dataset descriptions and search interfaces will facilitate dataset retrieval. Metadata analysis of selected data repositories Our study reveals that current metadata in data repositories match scholarly search interests only to some extent. Only if proper keywords are contained in general or descriptive fields, conventional retrieval algorithms are able to find relevant datasets. Poorly described data in combination with keyword-based retrieval techniques that are mainly used in retrieval systems

of data repositories [4] result in less relevant hits in search results or a long time spent until suitable data have been retrieved [3, 6, 7]. That might cause users to send explicit data requests to data repositories [5] or to switch to other data sources such as literature and general search engines [8]. Improved filter options such as a faceted search only support data seekers if either relevant, explicit metadata fields are provided or keywords are aligned and grouped [33]. Arbitrary metadata descriptions hamper both. Although metadata are not only created for search pur- poses, they are a key building block for successful data retrieval. Therefore, scholars need to be PLOS ONE Dataset search in biodiversity research PLOS ONE | https://doi.org/10.1371/journal.pone.0246099 March 24, 2021 29 / 36

aware that thorough data descriptions are part of a good scientific practice. In order to pre- serve all kind of scientific data, independently of whether it has been used in publications or not, proper metadata descriptions based on appropriate schemata enhance dataset retrieval and thus, increase data reuse and data citation. Data repositories could offer data curation ser- vices to support scholars in describing research data and to encourage them to describe their data thoroughly. We are aware that it would require high efforts to introduce more domain- specific metadata standards at generalist repositories; however, it would improve data descriptions. Challenges to be addressed in the biodiversity research community The importance of metadata for data discovery has already been recognized by the research community. In the FAIR Data Maturity Model [66], a guideline with indicators for the evalua- tion of the FAIR principles [11], all Findability indicators were considered as ‘essential’. The second Findability indicator (F2-01M) [66] refers to the richness of metadata. It says that an evaluation should consider the amount of metadata and its suitability for data discovery. How- ever, we think, it is unsatisfactory to tell scholars to provide ‘rich metadata’ without any further guidance, as it is often not clear what information is relevant for data discovery per research domain. Here, we recommend to extend the indicator, e.g., with additional documents per domain, providing more information about the various metadata standards and important

search categories per research field. Several studies already proposed guidelines for data reposi- tories [67] and scholars [68, 69] to ensure good data management favoring the usage of domain-specific standards. However, our study reveals that large data repositories only partly follow these recommendations. In particular, general data repositories tend to use only general metadata standards. Although guidelines and proper metadata data standards exist, the ques- tion remains why are they so rarely used? Are the recommendations too broad? Are there too many standards or do standards contain too many fields and elements? Do we need improved software tools and methods to facilitate the integration of different metadata in one search index? We think, only further user studies and discussions in the research community may help to find answers on these questions. First results of such discussions towards more fine-grained recommendations for essential metadata fields in specific research domains are the initiative from GBIF [70] and the work of Jurburg et al [71]. Both approaches aim for a better description of sequence data. We think, more in-depth discussions per research field will support data descriptions and metadata crea- tion as relevant search categories are domain dependent. In biodiversity research, a variety of standards are already available. Taxonomic and occurrence data could be comprehensively described with Darwin Core or ABCD. For environmental data descriptions, EML would be a