IOPS: computational graph optimization based on inter-operators parallel scheduling

To improve the inference efficiency of convolutional neural networks ( CNN) , the existing neu-ral networks mainly adopt heuristic and dynamic programming algorithms to realize parallel schedu-ling among operators. Heuristic scheduling algorithms can generate local optima easily, while the dynamic programming algorithm has a long convergence time for complex structural models. This pa-per mainly studies the parallel scheduling between operators and proposes an inter-operator parallel-ism schedule ( IOPS) scheduling algorithm that guarantees the minimum similar execution delay. Firstly, a graph partitioning algorithm based on the largest block is designed to split the neural net-work model into multiple subgraphs. Then, the operators that meet the conditions is replaced ac-cording to the defined operator replacement rules. Finally, the optimal scheduling method based on backtracking is used to schedule the computational graph. Network models such as Inception-v3, ResNet-50, and RandWire are selected for testing. The experimental results show that the algorithm designed in this paper can achieve a 1 . 6 × speedup compared with the existing sequential execution methods.