Enhancing Cloud-Based Big Data Analytics with Adaptive Job-Level Parallelism

Cloud computing has emerged as the preferred platform for executing big data analytics, primarily due to its inherent scalability, high availability, and cost-effectiveness. However, to fully leverage these benefits, efficient job scheduling is essential to ensure faster task execution and sustained cost-efficiency. This study aims to minimize the total processing time for batches of big data jobs without increasing the number of cloud servers. To achieve this, we propose an Adaptive Online Scheduling Algorithm that exploits parallelism at the job level, even in scenarios where job durations are unknown in advance. The proposed approach is evaluated against conventional two-phase and standard online scheduling methods using real-world datasets. Extensive simulation experiment confirm that our algorithm reduces job execution time by 25% as compared to two-phased algorithm and online scheduling algorithm with consistent and efficient performance, resulting in improved cloud resource utilization.