FlexSpark: Robust and Efficient Multi-Device Collaborative Inference over Wireless Network

The local deployment of large language models (LLMs) on mobile devices has garnered increasing attention due to its advantages in enhancing user privacy and enabling offline operation. However, given the limited computational resources of a single mobile device, only small language models (SLMs) with restricted capabilities can currently be supported. In this paper, we explore the potential of leveraging the collective computing power of multiple mobile devices to collaboratively support more efficient local LLM inference. We evaluate the feasibility and efficiency of existing parallelism techniques under the constraints of mobile devices and wireless network, identifying that chunked pipeline parallelism holds promise for realizing this vision. Building on this insight, we propose FlexSpark, a novel solution designed to achieve efficient and robust multi-device collaborative inference. FlexSpark incorporates priority scheduling, ordered communication, and elastic compression to maximize wireless bandwidth utilization, and thus accelerates distributed inference. Preliminary experimental results demonstrate that FlexSpark achieves up to a 2x speedup compared to state-of-the-art frameworks, significantly enhancing the practicality and scalability of LLM deployment on mobile devices.