Global Access to Legacy Data-Sets in Multi-cloud Applications with Onedata

Data access and management for multi-cloud applications proves challenging where there is a need for efficient access to large pre-existing, legacy data-sets. To address this problem, we created an indexing subsystem incorporated into Onedata data management system achieving a global multi-cloud integration of legacy storage systems containing large data-sets. The solution is based on metadata management, organization and periodic monitoring of legacy data-sets, what makes possible scheme-less co-existence a legacy storage system and Onedata for managing the same data collections. Thanks to block based data transfer provided by Onedata large files stored on legacy storage systems can be efficiently accessed from the cloud. The approach has been initially evaluated in a multi-cloud deployment scenario, where a large data collection stored on legacy storage system located in a super-computing center is processed on a commercial cloud.