TrainFlow: A Lightweight, Programmable ML Training Framework Via Serverless Paradigm

Distributed ML training is widely used to improve training performance. However, current distributed training frameworks bring undesirable burdens to application-oriented users due to its server-centric design. It is also difficult for users to customize training (e.g., with adaptive policies) to guarantee performance in dynamic environments. Thus, it is meaningful to make training framework lightweight and programmable. We argue that serverless paradigm can effectively help meet the demands. In this paper, we propose TrainFlow, adopting serverless paradigm to simplify and extend programmability of data-parallel training. First, the basic framework is built with a novel serverless process model, providing a high-level view and various state sharing. Then training can be divided into 2 processes with specific workflows. Second, TrainFlow provides an event-driven hook mechanism, allowing users to customize training workflow. We implement and evaluate TrainFlow with OpenFaaS. Experiments demonstrate its availability and programmability. For availability, TrainFlow can support various training patterns, and shows advantages of performance (e.g., 1.6× higher speedup ratio than baseline) and resource consuming (e.g., at most 41.0% less memory consuming than baseline). For programmability, TrainFlow can work with adaptive policies as expected (e.g., at most 1.48× higher throughput in a case).