MOTIVATION:Software plays a crucial and growing role in research. Unfortunately, the computational component in Life Sciences research is often challenging to reproduce and verify. It could be undocumented, opaque, contain unknown errors that affect the outcome, or be directly unavailable and impossible to use for others. These issues are detrimental to the overall quality of scientific research. One step to address this problem is the formulation of principles that research software in the domain should meet to ensure its quality and sustainability, resembling the FAIR (findable, accessible, interoperable, and reusable) data principles. RESULTS:We present here a comprehensive series of quantitative indicators based on a pragmatic interpretation of the FAIR Principles and their implementation on OpenEBench, ELIXIR's open platform providing both support for scientific benchmarking and an active observatory of quality-related features for Life Sciences research software. The results serve to understand the current practices around research software quality-related features and provide objective indications for improving them. AVAILABILITY AND IMPLEMENTATION:Software metadata, from 11 different sources, collected, integrated, and analysed in the context of this manuscript are available at https://doi.org/10.5281/zenodo.7311067. Code used for software metadata retrieval and processing is available in the following repository: https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL.

This report provides an overview of our activities and accomplishments concerning machine-actionable Software Management Plans (SMPs) and the Software Management Wizard (SMW) during the ELIXIR BioHackathon Europe 2023. ELIXIR acknowledges the critical role of effective software management in facilitating sustainable and reproducible research outcomes. The Software Best Practices group is actively committed to establishing a robust framework for SMP creation. In this project, our primary focus is on streamlining the SMP creation process for research software within ELIXIR. To achieve this, we are working on developing essential integrators and identifying and reviewing the relevant metadata schema. This effort is closely aligned with various related initiatives such as OpenEBench, FAIR4RS, RDA, maSMPs, among others. The outcomes of the BioHackathon project are now available for immediate use and can be further refined in the future based on community feedback and advancements in research software best practices.

Data Management Plans (DMPs) describe the data management lifecycle for the data corresponding to a research project, including activities from collection to preservation. Machine-actionable DMPs improve text-based DMPs by adding a semantic layer representing the most common elements relevant to DMPs, from datasets to funders. Similar to DMPs, Software Management Plans (SMPs) follow the software management lifecycle. The ELIXIR SMP was developed to support life science researchers and making it easier to follow research software good practices aligned to the findable, accessible, interoperable and reusable principles for research software. Currently, the ELIXIR SMP is a questionnaire-based document that would benefit from a machine-actionable approach. Here, we present a preliminary metadata analysis including possible types and properties from Schema.org that could be used to model machine-actionable SMPs.

Academic research requires careful handling of data plus any means to collect, transform and publish it, activities commonly supported by research software (from scripts to end-user applications). Data Management Plans (DMPs) are nowadays commonly requested by funders as part of good research practices. A DMP describes the data management lifecycle for the data corresponding to a research project, covering activities from collection to publication and preservation. To support and improve transparency, open science, reproducibility (and other *ilities), data needs to be accompanied by the software transforming it. Similar to DMPs, Software Management Plans (SMPs) can help formalize a set of structures and goals ensuring that the software is accessible and reusable in the short, medium and long term. DMPs and SMPs can be presented as text-based documents, guided by a set of questions corresponding to key points related to the lifecycle of either data or software. A step forward for DMPs are the machine-actionable DMPs (maDMPs) proposed by the Research Data Alliance DMP Common Standards Working Group. A maDMP corresponds to a structured representation of the most common elements present in a DMP (Miksa et al. 2020b), overcoming some obstacles linked to text-based representation. Such a structured representation makes it easier for DMPs to become readable and reusable for both humans and machines alike. The DMP Common Standard ontology (DCSO) (Cardoso et al. 2022) further supports maDMPs as it makes it easier to extend the original maDMP application profile to cover additional elements related to, for instance, SMPs or specific requirements from funders. maDMPs can be combined with the notion of a Research Object Crates (RO-Crate) to automate and ease management of research data (Miksa et al. 2020a). An RO-Crate (Soiland-Reyes et al. 2022) is an open, community-driven, and lightweight approach based on schema.org (Guha et al. 2016) annotations in JSON-LD to package research data (or any other research digital object) together with its metadata in a machine-readable manner. The ELIXIR SMP has been developed by the ELIXIR Software Development Best Practices Group in the ELIXIR Tools Platform to support researchers in life sciences (Alves et al. 2021). The ELIXIR SMP aims at making it easier to follow research software good practices aligned to the findable, accessible, interoperable and reusable principles for research software (FAIR4RS) (Chue Hong et al. 2022) while dealing with the lifecycle of research software. Its primary goal is encouraging a wider adoption by life science researchers, and being as inclusive as possible to the various levels of technical expertise. Here we present a roadmap for ELIXIR SMPs to become a FAIR digital object (FDO) (Schultes and Wittenburg 2019) based on the extension of maDMPs and DCSO and the use of RO-Crates. FDOs have been proposed as a way to package digital objects together with their metadata, types, identifiers and operations, so they become more machine-actionable and auto-contained. The current version of the ELIXIR SMP includes seven sections: accessibility and licensing, documentation, testing, interoperability, versioning, reproducibility, and recognition. Each section includes questions guiding and supporting researchers so they cover key aspects of the software lifecycle relevant to their own case. To lower the barrier and make it easier for researchers, most questions are Yes/No with some few offering a set of options. In some cases, a URL is also requested, for instance regarding the location of the documentation for end-users. Our roadmap for ELIXIR SMPs to move from a text-based questionnaire to an FDO comprises four main steps: creating maSMP application profile, extending DCSO, mapping to schema.org, and using RO-Crates. creating maSMP application profile, extending DCSO, mapping to schema.org, and using RO-Crates. Our maSMP application profile will include the semantic representation of the structured metadata that comes from the ELIXIR SMP. We will add granularity to the current root of the DCSO (dcso:DMP), by proposing the term SMP. In addition, we will propose the term ResearchSoftware as a dcso:Dataset. Terminology related to documentation, such as “Objective'' will also be considered. The objective is the Why the research software, which is crucial for their comprehensibility. We will propose the term DatasetObjective as the reason for the creation of a dataset. Source-codeRepository and Source-codeTesting are also good candidates to be part of the DCSO extension. We will extend DCSO with new classes and properties as necessary to include the software related elements mentioned in the maSMP application profile. As the ELIXIR SMP targets the life science community, we will analyze the need to add links from DCSO to ontologies describing common operations, activities, and types in this domain. One important aspect is the creation of a mapping from DCSO to schema.org. Schema.org has become a popular choice to add lightway semantics to web pages but can also be used on its own to provide metadata describing all sorts of objects. In life sciences, Bioschemas (Gray et al. 2017) offers guidelines on how to use some of the schema.org types aligned to this domain. Bioschemas includes a set of profiles, including minimum, recommended and optional properties, that have been agreed to and adopted by the community, for instance the ComputationalTool profile provides a way to describe software tools and applications. Bioschemas promotes its adoption by key resources in Life Sciences and development of tools such as the Bioschemas Markup Scraper and Extractor (BMUSE) used for the harvesting of the data (Gray et al. 2022). Our final step for ELIXIR SMPs to become an FDO is using RO-Crates to package research software together with its metadata and link it to/from its corresponding SMP. To do so, we will create an RO-Crate profile capturing the metadata needed to describe software tools including elements from the SMP. It will become a versioned living crate as research software evolves with time, particularly when new releases are published. Thanks to the RO-Crate bundling nature, where digital objects are packed together with its metadata, a software crate enriched with the elements from the SMP are a good example of an FDO as all the critical information about a software tool is bound together in a unit that can be shared with peers via FAIR registries and repositories.

Data dump of the OpenEBench Software Observatory, consisting in metadata of software in the Life Sciences domain extracted from several sources. All collections contain the same information at different points in the processing pipeline: - alambique collection contains metadata as extracted from sources. - pretools is the harmonized version of such metadata. - tools is the final integrated collection of software metadata used to identify trends and perform FAIRness evaluations. Repository containing the code used to generate the datasets: https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL

The FAIR Guiding Principles, published in 2016, aim to improve the findability, accessibility, interoperability and reusability of digital research objects for both humans and machines. Until now the FAIR principles have been mostly applied to research data. The ideas behind these principles are, however, also directly relevant to research software. Hence there is a distinct need to explore how the FAIR principles can be applied to software. In this work, we aim to summarize the current status of the debate around FAIR and software, as basis for the development of community-agreed principles for FAIR research software in the future. We discuss what makes software different from data with regard to the application of the FAIR principles, and which desired characteristics of research software go beyond FAIR. Then we present an analysis of where the existing principles can directly be applied to software, where they need to be adapted or reinterpreted, and where the definition of additional principles is required. Here interoperability has proven to be the most challenging principle, calling for particular attention in future discussions. Finally, we outline next steps on the way towards definite FAIR principles for research software.

The FAIR Guiding Principles were published and promoted to improve the reuse of scholarly data through making it more Findable, Accessible, Interoperable and Reusable by humans and machines. The intention was that the principles not only apply to data, but also to other digital objects, e.g. algorithms, tools, and workflows, that led to that data, as all these elements must be available to ensure transparency, reproducibility and reusability of research results. Over the last three years, numerous discussions have taken place that aimed to understand how the FAIR principles relate to software. A recent effort led to a systematic review and reinterpretation of the FAIR principles ( https://doi.org/10.3233/DS-190026 ), when applied to research software. Following on this collective effort, here we present our progress on the construction of a Software Quality Observatory for Life Sciences Research Software based on FAIR metrics, where software FAIR metrics of the bioinformatics tools universe will periodically measured and reported in an automatic way.

We understand benchmarking as the comparison of research software performance under controlled conditions. It encompasses both the technical performance, including software quality metrics, and the scientific performance in predefined challenges. Scientific communities play an important role here as they are responsible for defining reference datasets and metrics, pointing out the existing scientific challenges in their respective fields. Thus, in the context of ELIXIR-EXCELERATE project, we have developed the OpenEBench platform (https://openebench.bsc.es) aiming to provide a reference place to host technical and scientific performance for research software across the life sciences. OpenEBench provides an infrastructure where end-users can learn from different available software options and select the one best fitting their scientific needs. Bioinformatics software developers can find relevant datasets and meaningful scientific challenges to evaluate their own developments, and communities interested in a particular scientific domain can easily define which datasets and metrics are relevant for developers to work on, which in turn will allow the field to move ahead. A web application (https://openebench.bsc.es/html/scientific) allows users to browse through the benchmarking results from the different communities engaged (TCGA, QFO, CAMEO, GMI) in the platform which can be viewed using one of the visualization charts and transformed to table format, which is easier to interpret by non-expert users. On the technical side, the application uses several REST APIs, which can be used by other developers to upload and access the data for future studies or use cases. Another key feature is the Virtual Research Environment (https://openebench.bsc.es/submission), which includes the necessary mechanisms to import and execute benchmarking workflows on top of cloud computing infrastructures. OpenEBench follows the recommendations made by ELIXIR on the development of open source software making its code publicly available at https://github.com/inab/openebench-hub.