A secure data allocation solution for heterogeneous Hadoop systems: SecHDFS

Apache Hadoop is a widely used distributed computing framework and its file system is Hadoop Distributed File System (HDFS), which assumes that DataNodes in a system are homogeneous in nature. When a cloud system scales up, DataNodes are very likely to become heterogeneous. Thus, extensive research has been placed on improving performance for heterogeneous Hadoop systems, but little attention has been placed on security improvements. This motivates us to investigate a data allocation scheme called the Secure HDFS (SecHDFS) by integrating the secret sharing technique to improve storage security in a heterogeneous Hadoop system. DataNodes in a Hadoop system are classified into a variety of different types of groups based on their vulnerability characteristics. SecHDFS addresses the increased risk issue caused by data replication in HDFS by allocating fragments of a file to as many different types of DataNodes as possible and multiple replicas of the same fragment to DataNodes of the same type. A storage assurance model is developed to evaluate the quality of security offered by SecHDFS. Analysis of the assurance model and performance evaluation experiments show that SecHDFS does not impact the performance of the Hadoop system that much in comparison with the default HDFS, while significantly improving data assurance in a heterogeneous environment.