The National Research Data Infrastructure (NFDI) aims to create a sustainable and networked infrastructure for research data and helps to overcome the challenges associated with the storage, management and processing, security, and provision of research data in Germany [1]. It thus plays an important role in promoting open science and the exchange of FAIR research data. One of the NFDI initiatives is NFDI4Earth, which focuses on Earth System Sciences (ESS) [2]. Within the many ESS sub-disciplines, there is a diverse range of relevant high-quality data, services, tools, software, data repositories, as well as training and learning materials. Thus, it is not easy for researchers to find these various useful resources. Additionally, there is a lack of knowledge on how to use them due to an enormous diversity of standards, platforms, etc.The NFDI4Earth OneStop4All addresses these issues by serving as the primary user-friendly access point (Web portal) to the relevant ESS resources. It gives a coherent overview of the (distributed) resources for research data management (RDM), and data analysis/data science that are made available by the members of the NFDI4Earth as well as the Earth System Science (ESS) community. In particular, the OneStop4All provides access to data and software repositories, subject-specific RDM articles and a learning management system for open educational resources relevant to ESS researchers. In addition, it guides users through the NFDI4Earth resources according to their specific ESS RDM and data science needs and capabilities. The OneStop4All also promotes seamless access to a distributed user support network.The design and development of the OneStop4All is centered on the needs of the users. A good user experience requires an understanding of user behaviour, goals, motivations, and expectations and incorporating this knowledge into every stage of the design process. To achieve this, we use methods from user-centered design (UCD), complemented by knowledge and experience in various ESS disciplines from the members of the NFDI4Earth consortium, their extended scientific networks and by directly involving the community. We present the process of developing the user interface concept for the OneStop4All concerning usability and user experience and first insights into the platform are given. References[1] Agreement between the Federal Government and the Länder concerning the Establishment and Funding of a National Research Data Infrastructure (NDFI) of 26 November 2018: PDF-Datei [2] NFDI4Earth Consortium. (2022, Juli 7). NFDI4Earth - National Research Data Infrastructure for Earth System Sciences. Zenodo. https://doi.org/10.5281/zenodo.6806081
Governmental organizations collect and manage diverse data types at various levels to fulfill their official duties. This includes geographic, environmental, meteorological, population, health, traffic, transport, financial, and economic data. Traditionally, access to such data was restricted, but over the past decade, there has been a global shift towards more open data policies, influenced in part by directives like GeoIDG, the PSI directive, and INSPIRE. In Germany, federal states and their offices have also embraced open data policies, with some data being made publicly available (Open Government Data) through portals like Destatis or GDI-DE. This data serves multiple purposes, such as identifying locations, analyzing environmental trends, traffic planning, health service planning, and more. Public authorities' data is increasingly utilized for scientific investigations, yet the full potential remains untapped, particularly for large datasets. Despite the high quality of governmental data, further alignment with FAIR principles (Findable, Accessible, Interoperable, and Reusable) is necessary to enhance its efficiency for reuse in research. Privacy regulations and legal frameworks may impose limitations, necessitating data anonymization or adherence to modern data standards. Nevertheless, governmental data remains a valuable resource contributing significantly to expanding knowledge across scientific disciplines.In a pilot project funded by the NFDI4Earth, in a collaboration, the German national meteorological service (DWD) and the German Climate Computing Centre (DKRZ) aimed to facilitate access to data from public authorities, increase data visibility, as well as the number of users from different disciplines, and make these data available in standardised and FAIR formats for easy use in research but also for other public applications. As an example, the COSMO-REA6 reanalysis dataset from DWD (Kaspar et al. 2020) was selected, crucial for climate modeling, analyses, and energy applications in Europe. The standardization process involved mapping public authority standards to domain-specific standards in climate research, requiring close collaboration between DWD and DKRZ. After detailed curation and quality checks, the dataset was made accessible through the ESGF infrastructure and long-term archived in the WDCC, addressing licensing and authorship considerations.The project's insights and lessons learned were incorporated into a blueprint, providing guidance on making data from other authorities accessible and usable for both research and the public. Overall, the entire process can be divided into 5 sub-steps: (1) determination and classification of the need, (2) survey of the feasibility, (3) implementation, (4) feedback and follow-up, (5) dissemination. This blueprint outlines generalizable steps and aspects applicable across domains and collaborators, offering a framework for optimizing the use of governmental data in diverse fields.References: Kaspar, F., et al., 2020: Regional atmospheric reanalysis activities at Deutscher Wetterdienst: review of evaluation results and application examples with a focus on renewable energy, Adv. Sci. Res., 17, 115–128, https://doi.org/10.5194/asr-17-115-2020, 2020.
Climate science relies heavily on the effective creation, management, sharing, and analysis of massive and diverse datasets. As the digital landscape evolves, there is a growing need to establish a framework that ensures FAIRness in handling climate science digital objects. Especially, the machine-to-machine actionability of digital objects will be a crucial step towards future AI assisted workflows. Motivated by a use case, this contribution proposes the adoption of the Fair Digital Object (FDO) concept to address the challenges associated with the emerging spread in interdisciplinary reuse scenarios of climate model simulation output. The aim of this work is to commit to an FDO standard in climate science that enables standardized and therefore automated data analysis workflows and facilitates the extraction and analysis of relevant weather and climate data by all stakeholders involved. The current work expands on the efforts made to enable broad reuse of CMIP6 [1, 3] climate model data and focuses on requirements identified to enable automated processing of climate simulation output and their possible implementation strategies. The exemplary use case of an automated, prototypical climate model data analysis workflow will showcase the obstacles occurring when analyzing currently available climate model data. In particular, the findability of digital objects required for a particular research question in climate science or a related field shows to be challenging. In order to mitigate this issue, certain strategies are proposed: (a) Enriching the PID profiles of climate model data in accordance with the FDO concept and taking into account the needs of the climate science community will lead to improved findability of digital objects, especially for machines. (b) Defining a standardized, unique association between climate model variables and their meaningful long names will increase the findability of climate model data, especially for researchers in other disciplines. (c) Furthermore, combining the FDO concept with existing data management solutions, such as the intake-esm catalogs [2], can lead to improved data handling in line with prevailing community practices. Eventually, implementing an FDO standard will benefit the climate science community in several ways: The reusability of the data will facilitate the cost-effective use of existing computationally expensive climate model data. Improved data citation practices will promote data sharing, and ultimately, high transparency will increase the reproducibility of research workflows and consolidate scientific results.
This practice paper provides an overview of the Research Data Management Organiser (RDMO) software for data management planning and the RDMO community. It covers the background and history of RDMO as a funded project, its current status as a consortium and an open source software and provides insights into the organisation of the vibrant RDMO community. Furthermore, we introduce RDMO from a software developer’s perspective and outline, in detail, the current work in the different sub-working groups committed to developing DMP templates and related guidance materials.
The Special Collection Data Management Planning across Disciplines and Infrastructures of the Data Science Journal consists of papers describing practical experiences, concepts, and future directions on the design and deployment of effective data management plans and associated tools. Papers contain practical examples on managing and sharing data, consider the integration of data management plans into infrastructures and reflect innovative research into new directions for disciplinary and cross-disciplinary data management planning.
<p>Some disciplines, e.g. Astrophysics or Earth system sciences, work with large to very large amounts of data. Storing this data, but also processing it, is a challenge for researchers because novel concepts for processing data and workflows have not developed as quickly. This problem will only become more pronounced with the ever increasing performance of High Performance Computing (HPC) &#8211; systems.</p> <p>At the German Climate Computing Center, we analysed the users, their goals and working methods. DKRZ provides the climate science community with resources such as high-performance computing (HPC), data storage and specialised services and hosts the World Data Center for Climate (WDCC). In analysing users, we distinguish between two main groups: those who need the HPC system to run resource-intensive simulations and then analyse them, and those who reuse, build on and analyse existing data. Each group subdivides into subgroups. We have analysed the workflows for each identified user and found identical parts in an abstracted form and derived Canonical Workflow Modules. In the process, we critically examined the possible use of so-called FAIR Digital Objects (FDOs) and checked to what extent the derived workflows and workflow modules are actually future-proof.</p> <p>We will show the analysis of the different users, the Canonical workflow and the vision of the FDOs. Furthermore, we will present the framework Freva and further developments and implementations at DKRZ with respect to the reproducibility of simulation-based research in the ESS.</p>
Dataset from the Online Survey of the Research Data Alliance's Discipline-Specific Guidance for Data Management Plans Working Group.The data was collected from November 8, 2021 to January 14, 2022. The survey was divided into the following areas after a brief introduction on "Purpose of this survey" and "Use of the information you provide." Demographics Data Description and Collection Data Documentation & Quality Data Archiving, Publishing & Sharing After the Project Guidelines, Principles & Best Practices Follow-up interviews and group discussions The analysis of the online survey was focused on the four areas: Natural Sciences, Life Sciences, Humanities & Social Sciences, and Engineering. The results of the evaluation will be presented in a separate publication. In addition to the data, the variables and values are also published here. The online survey questions can be accessed here: https://doi.org/10.5281/zenodo.7443373 A more detailed analysis and description can be found in the paper "Discipline-specific Aspects in Data Management Planning" submitted to Data Science Journal (2022-12-15).
Economic risks for livestock production are caused by volatile commodities and market conditions, but also by environmental drivers such as increasing uncertainties due to weather anomalies and global warming. These risks impact the gross margin of farmers and can stimulate investment decisions. For confined pig and poultry production, farmers can reduce the environmental impact by implementing specific adaptation measures (AMs) to reduce heat stress. A simulation model driven by meteorological data was used to calculate indoor climate for 1981 to 2017. This dataset was extrapolated for a projection in the year 2030. The heat stress was calculated for a business-as-usual livestock building and for several AMs. For 2030, the expected value of the reduction of the gross margin was calculated by EUR 3.98 a−1 per animal place for the business-as-usual scenario. The results show that only energy-saving adaptation measures to reduce the inlet air temperature are appropriate to reduce the economic risk to the level of the year 1980 between EUR 0.03 a−1 and EUR 1.02 a−1 per animal place. The efficiency of other AMs to reduce heat stress is distinctly lower (between EUR 2.62 a−1 and EUR 3.80 a−1 per animal place). The results in this study can support the decision making of farmers concerning adaptation management and investments. It can inform agricultural policy design as well as technological development.
In addition to the previous intensive discussion on the “Data Deluge” with respect to enormous increase of available research data, the 2022 Internet-of-Things conference confirmed that in the near future there will be billions if not trillions of smart IoT devices in a very wide range of applications and locations, many of them with computational capacities. This large number of distributed IoT devices will create continuous streams of data that will require a global framework to facilitate their integration into the Internet to enable controlled access to their data and services, to name but a few aspects. This framework would enable tracking of these IoT devices to measure their resource usage for instance to globally address the UN Sustainable Development Goals. Additionally, policy makers are committed to define regulations to break data monopolies and increase sharing. The result will be an increasingly huge domain of accessible digital data which on the one hand allows addressing new challenges especially cross-sector ones. A key prerequisite for this is to find the right data across domain boundaries supporting a specific task. Digitisation is already being called the fourth industrial revolution and the emerging data and information is the 21st century's new resource. Currently this vision is mostly unrealised due to the inability of existing data and digital resources to be findable, accessible, interoperable, and reusable despite the progress in providing thematic catalogs. As a result, the capacity of this new resource is latent and mostly underutilized. There is no Internet level infrastructure that currently exists to facilitate the process by which all data and digital resources are made consistently and globally accessible. There are patchworks of localized and limited access to trusted data on the Internet created by specific communities that have been funded or directed to collaborate. To turn digital information into a commodity, description, access to, validation, and processing of data needs to become part of the Internet infrastructure we call the Global Integrated Data Space (GIDS). The main pillars of this approach require that data and services be globally identified and consistently accessed, with predictive descriptions and access control to make them globally findable. Currently researchers are relying partly on informal knowledge such as knowing the labs and persons to maximize the chance to access trustworthy data, but this method is limiting the use of suitable data. In the future data scenario, other mechanisms will become possible. In the public information space Google-like searches using specific key terms have become an accepted solution to find documents for human consumption. This approach however, does not work in the GIDS with large numbers of data contributors from a wide range of institutions, from millions of IoT devices worldwide, and where a wide range of data types and automatic data processing procedures dominate. Indeed, successful labs that apply complex models describing digital surrogates can automatically leverage data and data processing procedures from other labs. This makes the currently often operationally applied manual stitching of data and operations too costly both in time and resources to be a competitive option. A researcher looking for specific brain imaging data for a specific study has a few options: Rely on a network of colleagues. Execute Google-like searches in known registries looking for appropriate departments and researchers. Execute Google-like searches on suitable data. He/she engages an agent to execute profile matching in suitable sub-spaces. Rely on a network of colleagues. Execute Google-like searches in known registries looking for appropriate departments and researchers. Execute Google-like searches on suitable data. He/she engages an agent to execute profile matching in suitable sub-spaces. We assume that data creators will have the capability and be interested to create detailed metadata of different types and that the researchers, who are looking for specific data, will be able to specify precise profiles for data they are looking for. Two of the key characteristics of the future data space will be operations that can carry out profile matching at ultra-high speeds and that will lead to various subspaces according to some facets using self-organizing mechanisms. Of course, this poses high requirements on the metadata quality being used and that creators and potential consumers share knowledge about the semantic space in which they operate, and available semantic mappings used by brokers or self-provided. Metadata must be highly detailed and suitable schemas have been developed already in many communities. In addition to the usual metadata, potential users will need to specify their credentials in the form of trusted roles and their usage purposes to indicate access opportunities. Changing current metadata practices to yield richer metadata as presribed by the FAIR principles will not be simple, especially since we seem to be far away from formalizing roles and usage purposes in a broadly accepted way, but the pressure to create rich and standardized metadata will increase. It should be noted of course that for data streams created by IoT sensors, defining proper metadata is an action that is only requested once or a few times. Why are FDOs special in this automatic profile matching scenario? FDOs are bundling all information required for automatic profile matching in a secure way, i.e., all metadata information are available via the gloablly unique resolvable and persisten identifiers (PID) of the FDO and the PID security mechanisms are at the basis to establish trust. FDOs will be provided with a secure method that is capable of computing efficiently coded profiles representing all properties of an FDO relevant for profile matching. This would speedup profile matching enormously. We will address two major questions characterizing the “FDO Machine” we are envisioning: Which kinds of representations could make profile matching much more efficient? How could FDO-based mechanisms be used to efficiently create sub-spaces that would help the emerging layer of information brokers to offer specialized services addressing specialized needs as for example requested by UN’s Sustainable Development Goals? Which kinds of representations could make profile matching much more efficient? How could FDO-based mechanisms be used to efficiently create sub-spaces that would help the emerging layer of information brokers to offer specialized services addressing specialized needs as for example requested by UN’s Sustainable Development Goals? Brokers might want to use specialized agents to create subspaces along many different important facets such as domains, trustworthiness, roles, etc. Such subspaces are ephemeral virtual structures on top of the huge global integrated data space.
To provide tailored guidance on research data management, discipline-specific data management plan (DMP) templates are core. Different stakeholders are pursuing the development of such discipline-specific DMP templates in different ways. In this paper, we present three such approaches. First, we introduce the concept of domain data protocols, describing the work of the project Domain Data Protocols for Empirical Educational Research (in Germany). Members of this project develop tailored guidance on research data management (RDM) vertically, i.e., within the educational research discipline. Second, we present the approach of the Research Data Alliance Working Group on Discipline-specific Guidance for DMPs. The working group identifies discipline-specific DMP characteristics and uses them to analyse common practices and differences in RDM horizontally across disciplines. Finally, we describe the approach of the Research Data Management Organiser Sub-Working Group on Guidance, which facilitates mapping of discipline-specific DMP templates across disciplines. Although these three approaches differ in their underlying concepts, combining them facilitates identifying differences and commonalities in RDM across disciplines. By assigning discipline-specific characteristics and mapping these characteristics across disciplines, the three approaches improve our understanding of data management and research data within a particular discipline, as well as across disciplines.
This paper gives a summary of implementation activities in the realm of FAIR Digital Objects (FDO). It gives an idea which software components are robust and used for many years, which components
Motivation Results of simulations with climate models form the most important basis for research and statements about possible changes in the future global, regional and local climate. These output volumes are increasing at an exponential rate (Balaji et al. 2018, Stevens et al. 2019). Efficiently handling these amounts of data is a challenge for researchers, mainly because the development of novel data and workflow handling approaches have not proceeded at the same rate as data volume has been increasing. This problem will only become more pronounced with the ever increasing performance of High Performance Computing (HPC) - systems used to perform weather and climate simulations (Lawrence et al. 2018). For example, in the framework of the European Commission's Destination Earth program the Digital Twins (Bauer et al. 2021) are expected to produce hundreds of terabytes of model output data every day at the EuroHPC computing sites. The described data challenge can be dissected into several aspects, two of which we will focus on in this contribution. Available data in the Earth System Sciences (ESS) are increasingly made openly accessible by various institutions, such as universities, research centres and government agencies, in addition to subject-specific repositories. Further, the exploitability of weather and climate simulation output beyond the expert community by humans and automated agents (as described by the FAIR data principles (F-Findable, A-Accessable, I-Interoperable, R-Reusable), Wilkinson et al. 2016) is currently very limited if not impossible due to disorganized metadata or incomplete provenance information. Additionally, developments regarding globally available and FAIR workflows in the spirit of the FAIR Digital Object (FDO) framework (Schultes and Wittenburg 2019, Schwardmann 2020) are just at the beginning. Cultural Change In order to address the challenges with respect to data mentioned above, current efforts at DKRZ (German Climate Computing Center) are aimed at a complete restructuring of the way research is performed in simulation-based climate research (Anders et al. 2022, Mozaffari et al. 2022, Weigel et al. 2020). DKRZ is perfectly suited for this endeavor, because researchers have the resources and services available to conduct the entire suite of their data-intensive workflows - ranging from planning and setting up of model simulations, analyzing the model output, reusing existing large-volume datasets to data publication and long-term archival. At the moment, DKRZ-users do not have the possibility to orchestrate their workflows via a central service, but rather use a plethora of different tools to piece them together. Framework Environment Freva The central element of the new workflow environment at DKRZ shall be represented by the Freva (Free Evaluation System Framework) software infrastructure, which offers standardized data and tool solutions in ESS and is optimized for use on high-performance computer systems (Kadow et al. 2021). Freva is designed to be very well suited to the use of the FDO framework. The crucial aspects here are: the standardisation of data objects as input for analysis and processing, the already implemented remote access to data via a Persisitent Identifier (PID), the currently still system-internal capture of analysis provenance and the possibility of sharing results but also workflows by research groups up to large communities. the standardisation of data objects as input for analysis and processing, the already implemented remote access to data via a Persisitent Identifier (PID), the currently still system-internal capture of analysis provenance and the possibility of sharing results but also workflows by research groups up to large communities. It is planned to extend the functionality of Freva so that the system automatically determines the data required for a specific analysis from a researcher’s research question (provided to the system via some interface), enquires available databases (local disk or tape, cloud or federated resources) for that data and retrieves the data if possible. If data are not available (yet), Freva shall be able to automatically configure, set up and submit model simulations to the HPC-System, so that the required data is created and becomes available (cf. Fig. 1). These data will in turn be ingested into Freva’s data catalog for reuse. Next, Freva shall orchestrate and document the analysis performed. Results will be provided either as numerical fields, images or animations depending on the researcher’s need. As a final step, the applied workflow and/or underlying data are published in accordance with the FAIR data guiding principles. FDOs - towards a global integrated Data Space To make the process sketched out above a reality, application of the FDO concept is essential (Schwardmann 2020, Schultes and Wittenburg 2019). There is a long tradition in the ESS community of global dissemination and reuse of large-volume climate data sets. Community standards like those developed and applied in the framework of internationally coordinated model intercomparison studies (CMIP) allow for low-barrier reuse of data (Balaji et al. 2018). Globally resolvable PIDs are provided on a regular basis. Current community ESS standards and workflows are already close to being compatible with implementing FDOs, however, now we also have to work on open points in the FDO concept, which are: the clear definition of community-specific FDO requirements including PID Kernel Types specifications, the operation of data type registries and the technical implementation requirements for global access to FDOs. the clear definition of community-specific FDO requirements including PID Kernel Types specifications, the operation of data type registries and the technical implementation requirements for global access to FDOs. With these in place and implemented in Freva following standardized implementation recommendations, automated data queries across spatially distributed or different types of local databases become possible. We introduce the concept of implementations in Freva and also use it to highlight the challenges we face. Using an example, we show the vision of the work of a scientist in earth system science.
Some disciplines, especially those that look at the Earth system, work with large to very large amounts of data. Storing this data, but also processing it, places completely new demands on scientific work itself. Let's take the example of climate research and specifically climate modelling. In addition to long-term meteorological measurements in the recent past, results from climate models form the main basis for research and statements on past and possible future global, regional and local climate. Climate models are very complex numerical models that require high-performance computing. However, with the current and future increasing spatial and temporal resolution of the models, the demand for computing resources and storage space is also increasing. Previous working methods and processes no longer hold up and need to be rethought. Taking the German Climate Computing Centre (DKRZ) as an example, we analysed the users, their goals and working methods. DKRZ provides the climate science community with resources such as high-performance computing (HPC), data storage and specialised services and hosts the World Data Center for Climate (WDCC). In analysing users, we distinguish between two groups: those who need the HPC system to run resource-intensive simulations and then analyse them, and those who reuse, build on and analyse existing data. Each group subdivides into subgroups. We have analysed the workflows for each identified user and found identical parts in an abstracted form and derived Canonical Workflow Modules. In the process, we critically examined the possible use of so-called FAIR Digital Objects (FDOs) and checked to what extent the derived workflows and workflow modules are actually future-proof. The vision is that the global integrated data space is formed by standardised, independent and persistent entities that contain all information about diverse data objects (data, documents, metadata, software, etc.) so that human and, above all, machine agents can find, access, interpret and reuse (FAIR) them in an efficient and cost-saving way. At the same time, these units become independent of technologies and heterogeneous organisation of data, and will contain a built-in mechanism that supports data sovereignty. This will make the handling of data sustainable and secure. So, each step in a research workflow can be a FDO. In this case, the research is fully reproducible, but parts can also be exchanged and, e.g. experiments can be varied transparently. FDOs can easily be linked to others. The redundancy of data is minimised and thus also the susceptibility to errors is reduced. FDOs open up the possibility of combining data, software or whole parts of workflows in a new and simple but at all times comprehensible way. FDOs will make an important contribution to the reproducibility of research results, but they are also crucial for saving storage space. There are already data that are FDOs, but also self-contained frameworks that store data via tracking workflows. Similar to the TCP/IP standard, DO interface protocols are already developed. However, there are still some open points that are currently being worked on and defined with regard to FDOs in order to make them a globally functioning system.
The German Climate Computing Center (DKRZ) is an established topical IT service provider serving the needs of the German climate science community and their associated partners. At DKRZ, climate researchers have the means available to cover every aspect of the research life cycle, ranging from planning, model development and testing, model execution on the in-house HPC cluster (16 PFlops mainly CPU-based, 130 PB disk storage), data analysis (batch jobs, Jupyter, Freva), data publication and dissemination via the Earth System Grid Federation (ESGF) as well as long-term data preservation either at the project-level (little curation) or in the CoreTrustSeal certified World Data Center for Climate (WDCC) (extensive curation along the FAIR data principles). A plethora of user support services offered by domain-expert staff complement DKRZ’s portfolio. With the new HPC system coming online in early 2022 and a number of funded and to-be funded projects exploiting the available computational resources for conducting e.g. global storm-resolving (grid spacing O(1-3km)) simulations on climatic timescales, the current interplay DKRZ’s services needs to be revisited to devise a unified workflow that will be able to handle the upcoming challenges. This is why the above mentioned projects will supply a significant amount of funds to conceive a framework to efficiently orchestrate the entire model development, model execution and data handling workflow at DKRZ in close collaboration with the climate science community. In this contribution, we will detail our vision of a revamped and versatile ESM orchestration framework at DKRZ. Currently, this vision is based on having the orchestration performed by the Freva System (http://doi.org/10.5334/jors.253), in which users will be able to kick-off model compilation, compute and analysis jobs. Furthermore, Freva enables seamless provenance tracking of the entire workflow. Together with the implementation of data publication, long-term archiving and data dissemination workflows, the envisioned system provides a complete package of FAIR Digital Objects (FDOs) to researchers and allows for reproducibility, transparency and reduction of data redundancy.