Biodiversity informatics in an interdisciplinary field with a focus on organizing, accessing, provisioning and integrating a variety of sources of primary biodiversity data. What biodiversity informatics is not is a field devoted to developing statistical methodologies for analyzing biodiversity, but software development supporting ecological modeling is within it׳s purvey. Key topic areas in biodiversity informatics include: Community development of data and metadata standards to support provisioning of data and information conforming to FAIR data principles; Development of data publishing systems built on standards to assure interoperability; Development of data integration approaches to support integrating diverse data types ranging from co-locating resources to more sophisticated knowledge systems development and; methods and mechanisms to automate and enhance biodiversity data; and development of new means to extract data and information using tools such as natural language processing or other machine learning techniques. Together these approaches are meant to support community efforts to understand biodiversity and its change across the broadest phylogenetic, spatial and temporal scales.

BACKGROUND:The biodiversity domain, and in particular biological taxonomy, is moving in the direction of semantization of its research outputs. The present work introduces OpenBiodiv-O, the ontology that serves as the basis of the OpenBiodiv Knowledge Management System. Our intent is to provide an ontology that fills the gaps between ontologies for biodiversity resources, such as DarwinCore-based ontologies, and semantic publishing ontologies, such as the SPAR Ontologies. We bridge this gap by providing an ontology focusing on biological taxonomy.RESULTS:OpenBiodiv-O introduces classes, properties, and axioms in the domains of scholarly biodiversity publishing and biological taxonomy and aligns them with several important domain ontologies (FaBiO, DoCO, DwC, Darwin-SW, NOMEN, ENVO). By doing so, it bridges the ontological gap across scholarly biodiversity publishing and biological taxonomy and allows for the creation of a Linked Open Dataset (LOD) of biodiversity information (a biodiversity knowledge graph) and enables the creation of the OpenBiodiv Knowledge Management System. A key feature of the ontology is that it is an ontology of the scientific process of biological taxonomy and not of any particular state of knowledge. This feature allows it to express a multiplicity of scientific opinions. The resulting OpenBiodiv knowledge system may gain a high level of trust in the scientific community as it does not force a scientific opinion on its users (e.g. practicing taxonomists, library researchers, etc.), but rather provides the tools for experts to encode different views as science progresses.CONCLUSIONS:OpenBiodiv-O provides a conceptual model of the structure of a biodiversity publication and the development of related taxonomic concepts. It also serves as the basis for the OpenBiodiv Knowledge Management System.

Ontology is a plain text file containing statements in the Turtle syntax forming OpenBiodiv-O. It can be edited in a text (e.g. Sublime Text, Emacs, etc.) or in an ontology editor (e.g. ProtĂŠgĂŠ). It can be loaded it into a triple store (e.g. GraphDB). The prefixes that are used throughout this manuscript are defined at the beginning. This file corresponds to < http://openbiodiv.net/openbiodivo-20171103 >. (TXT 22 kb)

Since its ratification as a TDWG standard in 2009, data publishers have had to struggle with the essential step of mapping fields in working databases to the terms in Darwin Core Wieczorek et al. 2012 in order to publish and share data using that standard. Doing so requires a good understanding of both the data set and Darwin Core. The accumulated knowledge about these mappings constitutes what we call the "Darwin Cloud." We will explore the nature of data mapping challenges and the potential for semi-automated solutions to them. Specifically, we will look at the "Darwinizer" actor and its usage in related workflows within the Kurator data quality framework and the implications for community-managed vocabularies.

In the Kurator project, we are developing libraries of small modules, each designed to address a particular data quality test. These libraries, which can be run on single computers or scalable architecture, can be incorporated into data management processes in the form of customizable data quality scripts. A script composed of these modules can be incorporated into other software, run as command-line programs, or provided as a suite of “canned” workflows through a web interface. In some of these modules, we have implemented a subset of the standard tests under development by Task Group 2 (TG2) of the Data Quality Interest Group. We have also been exploring use of the fitness-for-use-framework (Veiga et al. 2017) produced by Task Group 1 (TG1) of the Data Quality Interest Group. Our goals have been to explore use of the framework to describe capabilities of atomic modules of code, how we can use concepts in the framework to produce data quality reports, and what lessons can be learned from implementing data quality control code in the context of the framework. We have focused on the Data Quality Reports level of the framework; in particular, the representation of Data Quality Measures (measurements on some data quality dimension), ‡ ‡ ‡,§ | ¶ | §,‡ ‡ | © Morris P et al. This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Validations (tests for compliance with quality needs), and Amendments (proposals to improve data quality). At the implementation level, we have developed a set of Java annotations to mark methods as providing specific tests from the test suite. In terms of the framework, the annotations can also be used to mark methods as providing Measures, Validations, or Amendments and to associate method parameters with Information Elements by linking them to the Darwin Core terms that were either "acted upon" or "consulted" (Lowery et al. 2016). These annotations can be used by a consumer to identify and run Measures and Validations in two phases: a Pre-Amendment phase, before the Amendments are run; and a Post-Amendment phase, after the changes proposed by the Amendments have been applied. Capturing the test results across both stages allows us to report on how much accepting the amendments would improve the quality of the dataset as a whole, data in some quality dimension, or data for some specific purpose. We have found it important to be able to render data quality reports that identify which Darwin Core terms are the Valuable Information elements involved in a specific test, and, further, to identify which terms are acted upon and which are consulted. Identifying this information allows us, for example, to render tabular reports highlighting cells where amendments have proposed a change. We have also found reporting of error and failure conditions to be important, and have been working on implementing the TG1 suggestion that report elements consist of a result (containing only appropriate values), status (containing a controlled vocabulary term such as completed, or data_prerequisites_not_met), and a human readable message (metadata about why the conclusion that was drawn was drawn, or error messages). We have developed a stake-in-the-ground vocabulary for status values to describe failure conditions including the following concepts: Ambiguous (there was a result, but it has ambiguity, e.g., an event date inferred from the verbatim event date 04/05/1954), Internal Prerequisites Not Met (not able to run the test on the data provided, e.g., day was not an integer), and External Prerequisites Not Met (some external resource that this test consults was unavailable at runtime). In implementing tests in the context of the framework, we have seen the value of identifying Measures, Validations, and Amendments in forcing us to develop small, focused tests, and in allowing us group assertions within data quality reports based on data quality needs.

The YesWorkflow McPhillips et al. 2015b, McPhillips et al. 2015a toolkit was designed to annotate data curation workflows in conventional scripts (e.g., Python, R, Java) but it can also be used to annotate YAML-based Kurator workflow configuration files. From just a file that has been annotated by YesWorkflow, YesWorkflow is able to render a top-level graphical view of the workflow structure (prospective provenance), including system inputs and outputs, actors, connections among those actors, and expected data to be passed on those connections. YesWorkflow also supports dynamic analysis and reporting on the results of the workflow (retrospective provenance) at various levels of granularity (e.g., at the actor level, script level, data level, record level, file level, function level), provided that it has been configured at each. YesWorkflow includes an @Log annotation, which describes the semantic structure of a log message within some actor in the workflow and allows the log message to be linked to the actor within which it was created, and for parts of that log message to be linked to the data passed between actors. YesWorkflow can be used to analyze the log messages after a run of the workflow and construct a store of facts, which can be queried and reasoned upon to make statements about the evolving paths taken by particular data elements through the workflow and assertions made about those data elements within the workflow. Provenance, like other metadata, appears to be rarely actionable or immediately useful for those who are expected to provide it. However, by refactoring and integrating runtime observables generated from retrospective provenance and context information from prospective provenance analysis into hybrid queries, we show how both elements can yield hybrid visualizations that reveal “the plot” of the whole execution. In this way, a comprehensive workflow graph and a customizable data lineage report are made actionable for a workflow run with meaningful provenance artifacts. Queries run on a set of facts extracted from log messages by YesWorkflow after a workflow run, in combination with the facts extracted from the annotated workflow itself, allow for powerful visualizations of the retrospective provenance of a workflow run and of particular data records within a branching workflow.

Data cleaning has the potential to improve the chances for people and computers to find and use relevant data. This is true for researchers as well as for large-scale data aggregators. In the biodiversity realm, Darwin Core provides a convenient scope and framework for data cleaning tools and vocabularies. One way to address data cleaning tasks is to use workflows that act on a combination of original data, controlled vocabularies, algorithms, and services to detect inconsistencies and errors, recommend changes, and augment the original data with improvements and additions. There are advantages from the perspective of flexibility to construct such workflows from specialized, reusable -- building blocks that do specific tasks, such as provide a list of distinct values of a field in a data set. The Kurator project uses Akka, a Java-based framework to construct workflows with actors written in a variety and even in a combination of programming languages. In this presentation, we will explore the process of building actors and combining them in Akka workflows that do a variety of data cleaning and reporting tasks inspired by the VertNet process of mobilizing data from institutional data sets for large-scale aggregators such as VertNet, iDigBio, and the Global Biodiversity Information Facility. Ultimately, the goal of this work might be, given a biodiversity data set, to provide an improved version of that data set in the form of a Darwin Core archive that includes a data quality extension (not yet developed) to report what was found, what was done to it, and what could still be done to further improve it.

Background: Taxonomic descriptions are traditionally composed in natural language and published in a format that cannot be directly used by computers. The Exploring Taxon Concepts (ETC) project has been developing a set of web-based software tools that convert morphological descriptions published in telegraphic style to character data that can be reused and repurposed. This paper introduces the first semi-automated pipeline, to our knowledge, that converts morphological descriptions into taxon-character matrices to support systematics and evolutionary biology research. We then demonstrate and evaluate the use of the ETC Input Creation - Text Capture - Matrix Generation pipeline to generate body part measurement matrices from a set of 188 spider morphological descriptions and report the findings.Results: From the given set of spider taxonomic publications, two versions of input (original and normalized) were generated and used by the ETC Text Capture and ETC Matrix Generation tools. The tools produced two corresponding spider body part measurement matrices, and the matrix from the normalized input was found to be much more similar to a gold standard matrix hand-curated by the scientist co-authors. Special conventions utilized in the original descriptions (e.g., the omission of measurement units) were attributed to the lower performance of using the original input. The results show that simple normalization of the description text greatly increased the quality of the machine-generated matrix and reduced edit effort. The machine-generated matrix also helped identify issues in the gold standard matrix.Conclusions: ETC Text Capture and ETC Matrix Generation are low-barrier and effective tools for extracting measurement values from spider taxonomic descriptions and are more effective when the descriptions are self-contained. Special conventions that make the description text less self contained challenge automated extraction of data from biodiversity descriptions and hinder the automated reuse of the published knowledge. The tools will be updated to support new requirements revealed in this case study.

Data cleaning has the potential to improve the chances for people and computers to find and use relevant data. This is true for researchers as well as for large-scale data aggregators. In the biodiversity realm, Darwin Core provides a convenient scope and framework for data cleaning tools and vocabularies.

In the FilteredPush (FP) and Kurator projects we have built tools for quality control of biodiversity data. One of these, FP-Akka , is derived from earlier work on the Kepler Kuration package, where, to run data-curation workflows within a FilteredPush node infrastructure, code was refactored into an external service wrapper layer, a data validation logic layer, and a workflow layer that composes elements of the logic layer into actors in a record-centric workflow. The service wrapper layer and data validation layer are packaged in an FP-KurationServices library, which can be composed with Kepler Kuration workflows or with workflows written in the Akka parallelization framework. In developing and maintaining FP-Akka, we encountered multiple challenges arising from the interplay between external services and workflow components: discovery of pertinent services, technical documentation and integration of services, documentation of domain-specific details of assumptions made by the services, the wide variety of technologies used by service providers, and maintenance of our code base in the face of changing services. To find services pertinent to the data-quality needs of the science goals of US Thematic Collections Networks (TCNs)—principally, quality control of scientific names, georeferences, and collecting event dates—we have looked, ad hoc, to the usual suspects for quality data in relevant domains and have done some discovery using The Biodiversity Catalogue service registry. We also wrote a service (for name and date data concerning entomologists) in a case where shallow searching for pertinent services returned no results. Service documentation at the technical level has ranged from Web Services Description Language (WSDL) files from which we could generate code, to example response documents, to none; in the latter case, we simply coded to the observed responses of the service. Much more difficult has been domain concepts documentation, the information needed to tell what assumptions the service provider is making about questions made to the service, and what assumptions are embedded in the responses. In a simple case, such as for a service that provides information related to scientific names, are the responses from the service making nomenclatural assertions, taxonomic assertions, or a mixture of both? Also relevant to understanding of service use is how clean and authoritative is the dataset behind the service, and, when quality varies in a dataset, presence of row-level assertions about data quality. Biodiversity-related service implementations use a wide variety of exchange technologies, in effect requiring consumers to do something different for each service we interact with, thus each component in our service-wrapper layer is wholly different from the others. The combination of different technologies for services and different domain assumptions made by similar services have combined to make it difficult for us to cleanly write a layer containing our validation logic and compose this with a layer that abstracts services. Instead, we have had to bring some of the logic dealing with the differing assumptions of different services down into the service-wrapper layer. To maintain the code, we responded to both documented and silent changes to service Application Programming Interfaces (APIs). In order to detect API changes that would cause our code to fail, we have unit tests that query the services. This has created challenges for our development framework, as when a service has a transient outage, builds that run these tests fail (and, in the case of automated build systems, tend not to leave clear traces of failure causes). Our single largest challenge has not been technological, but social: to understand the domain-specific assumptions of biodiversity data service providers.

The Kurator project aims to facilitate the development, documentation, and efficient execution of scripts and workflows for cleaning biodiversity data. Kurator tools under development and available as prototypes in the Kurator GitHub repositories ( http://github.com/kurator-org/ ) support traditional scripting as well as high-performance, actor-oriented workflow approaches to validating, annotating, and cleaning data. The Kurator-Akka framework ( http://github.com/kurator-org/kurator-akka ) makes it easy to develop and run high-performance data cleaning workflows that employ the Akka actor toolkit by shielding actor developers and workflow users alike from the complexities of the Akka API (application programming interface). Kurator-Akka actors currently can be written either in Python or Java, and workflows may be specified using a language based on YAML (YAML Ain't Markup Language) that defines how data flows between the actors at run time. A workflow can be composed from existing actors by editing a simple text file and subsequently executed by providing this file to the Kurator-Akka workflow runtime. Actors in a Kurator-Akka workflow execute concurrently in different threads, potentially yielding pipeline parallelism and thus higher throughput than is achievable in conventional scripts. Recognizing that conventional scripts written, for example, in Bash, Python or R, also represent an effective means of automating data cleaning workflows, Kurator is leading an interdisciplinary effort to develop the YesWorkflow toolkit ( http://yesworkflow.org/yw ). YesWorkflow (YW) aims to provide many of the benefits of using a scientific workflow management system without having to rewrite scripts for execution within a workflow engine. Instead, a YesWorkflow user simply adds special YesWorkflow comments to existing scripts. These comments declare how data is used and results produced, step by step, by the script. The YesWorkflow tools interpret the YW comments and produce graphical output that reveals the stages of computation and the flow of data in the script. A means for reconstructing and querying the provenance of the outputs of a script marked up with YesWorkflow annotations is currently under development. Because scripts marked up with YesWorkflow annotations may be used as actors in the Kurator-Akka framework, Kurator tools span the scripting and workflow automation paradigms. The Kurator-Validation GitHub repository ( http://github.com/kurator-org/kurator-validation ) provides example scripts, actors, workflows, and documentation that demonstrate how Kurator tools effectively integrate scripting and automated workflow approaches to cleaning biodiversity data. Using a simple Python class that wraps the WoRMS (World Register of Marine Species) web service as a starting point, documentation available in the Kurator-Validation GitHub repository demonstrate (1) how a Python script can make use of this WoRMS service class to validate names against the standard WoRMS taxonomy; (2) how to annotate this script with YesWorkflow comments so the script can be modeled, visualized and analyzed as a workflow; (3) how a Kurator-Akka actor invoking the WoRMS service class can be written in Python; and (4) how a workflow employing this actor can be specified in YAML and executed by the Kurator-Akka framework with each actor running concurrently.

Natural Science collections tend to be highly organized for storage and retrieval having material with the same current identification stored together and often secondarily segregated by geography. The information content associated with the storage system can be exploited to improve digitization efficiency. In object-to-image-to-data workflows, imaging of specimen label data separates physical specimen handling from the capture of data associated with the specimen. If information about the storage of the collection can be carried through the imaging step into data records, it is possible to capture inventory-level information about the collection prior to imaging, and to associate that information with images. In the New England Vascular Plant (NEVP) Thematic Collections Network (TCN), and in digitization projects in the entomology collections in the Museum of Comparative Zoology (MCZ), an inventory level pre-capture pass is performed in collections. The current identification of specimens is printed out, in machine-readable form, associated with the physical storage units (unit trays or folders), and then processed in the imaging step to create skeletal data records to accompany the image. In both cases, structured data is stored in JSON in a QR-code 2d barcode, produced by the MCZ's open source DataShot software, which uses the ZXing library for barcode generation and reading. The MCZ entomology projects exploit the organization of the collection into unit trays, which contain material sharing a current identification. A machine-readable label containing the current identification (and often the drawer number) is printed out for each unit tray. During imaging, an individual specimen is removed from the unit tray, its labels are removed, and the specimen and its labels are placed on a jig. The machine-readable label containing the current identification for the unit tray is included in the image, as is a machine-readable label containing the catalog number for the specimen. The imaging step involves only specimen handling and imaging, and no data capture, but machine processing of the image creates skeletal database records that contain the current identification, drawer number, and catalog number. Processing of machine-readable data in the image allows decoupling of capture of information inherent in the storage, specimen handling, and electronic data creation. A similar pass through the collections has been performed in the NEVP TCN, printing out the current identification (and often geographic information) for material in a folder in machine-readable form. During the imaging and handling step, at a primary digitization apparatus (which may not be co-located with the collection), this information is captured with a barcode scanner, as is the barcode/catalog number of the specimen. In addition, locality, collector name, collector number, and collection date are transcribed by an operator of the primary digitization apparatus during the imaging step. This creates a minimal data record of scientific name, town in which the material was collected, and the date it was collected. This information is then encoded in an OA annotation for transfer to consuming systems (Symbiota, and the database of record), allowing decoupling of specimen handling and skeletal record creation from database systems.

Botanists intentionally collect multiple specimens from the same organism and distribute them to several collections for curation. During or after the initial digitization of legacy paper occurrence metadata, the resulting digital records may diverge for a number of reasons. Some of this divergence results from differing curation velocities, eg, one duplicate may receive a taxonomically current species assignment while another does not. Other divergence may arise as a simple consequence of human transcription error. Yet other may reflect different local curatorial practices and not be regarded as an error at all; for example, one collection may record the collector’s name in a single field, first name followed by last, whereas another collection might do the reverse. There are similar issues about legacy event dates. These “non-erroneous” divergences, whether or not they involve duplicates, can impact the fitness of …

Data curation is increasingly important. Our previous work on a Kepler curation package has demonstrated advantages that come from automating data curation pipelines by using workflow systems. However, manually designed curation workflows can be error-prone and inefficient due to a lack of user understanding of the workflow system, misuse of actors, or human error. Correcting problematic workflows is often very time-consuming. A more proactive workflow system can help users avoid such pitfalls. For example, static analysis before execution can be used to detect the potential problems in a workflow and help the user to improve workflow design. In this paper, we propose a declarative workflow approach that supports semi-automated workflow design, analysis and optimization. We show how the workflow design engine helps users to construct data curation workflows, how the workflow analysis engine detects different design problems of workflows and how workflows can be optimized by exploiting parallelism.

Biological specimens in research collections provide the most important baseline information for systematic research. Traditionally, they are annotated by experts in written form, which remains directly associated with the specimens. These annotations, defined as data added at a later stage to the original data, provide an important quality control mechanism. They improve the value of herbarium specimens and are identification trails documenting the development of taxonomic concepts over time. With specimen data increasingly becoming accessible via the Internet, a general online annotation system that ensures that the traditional data sharing and documentation of specimen data is continued after the information is mobilised through digitisation, is currently missing. We lay out the prerequisites for such an annotation system including data standards, a data repository, system access, and user roles. We also introduce an exemplar solution developed in the DFG-funded AnnoSys project. AnnoSys is being implemented using the example of collection and observation data in the botanical domain as provided by the GBIF/BioCASe networks. It provides a user-friendly interface to allow researchers to produce and discover annotations. If a record has been annotated, both the annotation and the original record will be stored in a repository, linked via a persistent identifier, and will be accessible through the AnnoSys interfaces. Collection holders and scientists specifically interested in a subset of data will be informed about annotations in which they have expressed interest. We discuss AnnoSys in relation to the Filtered Push project, which pursues the same goal in facilitating and communicating online annotations, but which takes a different approach.

The Audubon Core Multimedia Resource Metadata Schema is a representation-free vocabulary for the description of biodiversity multimedia resources and collections, now in the final stages as a proposed Biodiversity Informatics Standards (TDWG) standard. By defining only six terms as mandatory, it seeks to lighten the burden for providing or using multimedia useful for biodiversity science. At the same time it offers rich optional metadata terms that can help curators of multimedia collections provide authoritative media that document species occurrence, ecosystems, identification tools, ontologies, and many other kinds of biodiversity documents or data. About half of the vocabulary is re-used from other relevant controlled vocabularies that are often already in use for multimedia metadata, thereby reducing the mapping burden on existing repositories. A central design goal is to allow consuming applications to have a high likelihood of discovering suitable resources, reducing the human examination effort that might be required to decide if the resource is fit for the purpose of the application.

A scientific workflow describes a process for accomplishing a scientific objective, usually in terms of tasks (implemented by software components, or actors) and their dataflow dependencies. Scientific workflows have become an increasingly popular paradigm and tool in many eSciences and can be used to improve various computation-intensive and data-intensive processes in biodiversity informatics.

Electronic annotation of scientific data is very similar to annotation of documents. Both types of annotation amplify the original object, add related knowledge to it, and dispute or support assertions in it. In each case, annotation is a framework for discourse about the original object, and, in each case, an annotation needs to clearly identify its scope and its own terminology. However, electronic annotation of data differs from annotation of documents: the content of the annotations, including expectations and supporting evidence, is more often shared among members of networks. Any consequent actions taken by the holders of the annotated data could be shared as well. But even those current annotation systems that admit data as their subject often make it difficult or impossible to annotate at fine-enough granularity to use the results in this way for data quality control. We address these kinds of issues by offering simple extensions to an existing annotation ontology and describe how the results support an interest-based distribution of annotations. We are using the result to design and deploy a platform that supports annotation services overlaid on networks of distributed data, with particular application to data quality control. Our initial instance supports a set of natural science collection metadata services. An important application is the support for data quality control and provision of missing data. A previous proof of concept demonstrated such use based on data annotations modeled with XML-Schema.