This project outcome is allied to the Data and Digital Output Management Plan for the project (Stall et al. 2023). This resource contains templates for others to use when tracking their project outcomes and usages, accompanied by filled worksheets for the PARSEC project up until the date of this version (19 August 2023). The material provided is available as examplars from the PARSEC project and for use by others (excel templates) for the following categories: Scholarly Publications Non Peer-Reviewed Digital Outputs (to include Datasets, Software, Conferences, Posters, Presentations, Training Materials, Workshop Materials, etc.) Datasets - working log of datasets reviewed for use by the project Software - working log for software reviewed by the project (to include Models, Notebooks, Workflow, etc.) Stall, Shelley, Specht, Alison, Corrêa, Pedro Luiz Pizzigatti, David, Romain, Edmunds, Rorie, Mabile, Laurence, Machicao, Jeaneth, Miyairi, Nobuko, Murayama, Yasuhiro, O'Brien, Margaret, Wyborn, Lesley, & Vellenich, Danton Ferreira. (2023). PARSEC Data and Digital Output Management Plan and Workbook. Zenodo. https://doi.org/10.5281/zenodo.3891426.
The challenges of Reproducibility and Replicability (R & R) in computer science experiments have become a focus of attention in the last decade, as efforts to adhere to good research practices have increased. However, experiments using Deep Learning (DL) remain difficult to reproduce due to the complexity of the techniques used. Challenges such as estimating poverty indicators (e.g., wealth index levels) from remote sensing imagery, requiring the use of huge volumes of data across different geographic locations, would be impossible without the use of DL technology. To test the reproducibility of DL experiments, we report a review of the reproducibility of three DL experiments which analyze visual indicators from satellite and street imagery. For each experiment, we identify the challenges found in the data sets, methods and workflows used. As a result of this assessment we propose a checklist incorporating relevant FAIR principles to screen an experiment for its reproducibility. Based on the lessons learned from this study, we recommend a set of actions aimed to improve the reproducibility of such experiments and reduce the likelihood of wasted effort. We believe that the target audience is broad, from researchers seeking to reproduce an experiment, authors reporting an experiment, or reviewers seeking to assess the work of others.
