Approximate Gradient Coding and Data Assignment in Distributed Computing Systems

In the deployment of distributed gradient descent algorithms in a network, the computational time and response time of a worker node are affected by various factors, such as processor speed, memory, network delay, and congestion. Straggler feedback delays can severely set back the effectiveness of distributed learning. To leverage the advantage of parallel processing in model training, gradient coding is designed to mitigate the effect of stragglers. This paper investigates gradient coding for heterogeneous workers with varying computational capabilities. We formulate the problem of approximating the gradient vector by minimizing the average error of the recovered gradient vector and propose a solution that leverages fractional repetition codes for data assignment to mitigate the impact of stragglers.