An open science platform for benchmarking engineering design researches

In the last fifteen years, several studies showed that a large part of research in engineering design fails to provide objective evidence demonstrating real progress because of a lack of a generally accepted and effective validation environment to test, confirm or refute results. A better validation protocol would highlight the scientific added value on socio-technical systems, facilitate an incremental construction of solutions and improve the recognition of the engineering design research community. Thus, an open science environment for benchmarking engineering design research is proposed. First, a set of guidelines that helps researchers to define the minimum viable content of a benchmark is introduced. Second, the implementation of these guidelines is described in a web-based version-control and collaborative platform within which a community of researchers can engage in the building of a sustainable validation ecosystem. Future works will concentrate on the consolidation of the platform (benchmark examples, instructions, workflows, licenses, etc.) based on the feedback of the current members of the French S.mart special interest group (s-mart.fr) before advertising to a larger community of researchers.