Task-load Aware and Predictive-Based Workflow Scheduling in Cloud-Edge Collaborative Environment

There are a large number of real-time streaming data task processing workflows in online anomaly detection and intelligent transportation systems. Real-time data transmission is required between these processing tasks. To ensure real-time performance, it is inevitable for real-time workflow scheduling. However, due to the connected sensor data being updated in real-time, the task load of the workflow will fluctuate. In this case, the original scheduling plan is no longer optimal and needs to be renewed. Adjustments in scheduling plan bring new challenges to dynamic scheduling. At the same time, when the scheduling plan changes, the deep learning models and data used in these workflow tasks need to be redeployed at a high cost. To address these challenges, we use the data generating rate to express the task load’s fluctuation. Besides, we also consider the migration cost caused by the adjustment of the scheme in the continuous scheduling process during modeling and optimization. To achieve a low-cost scheduling scheme in a short time, we propose the NSGA-II-Seq2Seq algorithm which employs the historical scheduling plan to generate a candidate scheduling plan. The experiments are carried out in the modified WorkflowSim. Through experiment, it is found that the method proposed in this paper can adapt to the change of task load and can produce an adaptively fine-tuning scheduling scheme. In multiple consecutive scheduling experiments, the population optimization process of NSGA is accelerated, which significantly reduces the time to obtain the scheduling plan.