EdgeInferFlow: A Distributed Inference Acceleration Method for Deep Learning Chained Structure Models for Edge Devices.

With the increasing deployment of complex deep learning models on edge devices, addressing the high inference latency caused by their inherent computational bottlenecks is paramount for enabling real-telligent applications. To mitigate the resulting high inference latency, we propose EdgeInferFlow, a novel framework designed to accelerate the distributed inference of chain-structured models across edge-device clusters. The framework is predicated on a two-level optimization methodology: model partitioning and distributed inference orchestration. At the partitioning level, EdgeInferFlow employs a dynamic load-balancing algorithm to mitigate computational load imbalance among partitioned sub-models. This algorithm is integrated with a distributed computation scheme for linear layers and a multi-stage strategy to minimize inter-node data dependencies and communication overhead. At the inference level, EdgeInferFlow constructs a distributed computational graph to orchestrate the data flow and parallel execution of tensors throughout the cluster. Experimental results demonstrate that our proposed method achieves a reduction in end-to-end inference latency of up to 70