A Utility-Based Distributed Pattern Mining Algorithm with Reduced Shuffle Overhead

With the arrival of the current digital era and the advancement of information transmission technologies, there has been an unprecedented rise in data. Efficient extraction of useful information from the volumes of data has garnered growing interest from academics and the industry. Data mining research focuses on finding utility patterns in large datasets. But the inherent complications like frequent scans, creation of substantial candidate sets, etc. plague the mining process for large datasets. Distributive architecture-based approaches also prove inefficacious due to high communication overhead over iterations. High communication cost over data exchange both locally and remotely further aggravates the situation. We propose a Communication Cost Effective Utility-based Pattern Mining (CEUPM) algorithm based on the Spark framework to address this issue. Spark accelerates iterative scanning by storing scanned datasets in a memory abstraction called resilient distributed datasets (RDD). RDD operations need a redistribution of data among cluster nodes during processing. To minimize the communication cost incurred during the shuffle process, we adopt a search space division strategy based on data parallelism for a fair and effective task allocation across cluster nodes. Communication overhead is incurred during this redistribution or shuffle process while minimizing costs. Experimental results in four real datasets demonstrate that CEUPM considerably reduces shuffling overhead and outperforms other existing methods in terms of memory usage, communication cost, execution time, and scalability.