Maximization and restoration: Action segmentation through dilation passing and temporal reconstruction

•We propose a divide-and-conquer method that first maximizes frame accuracy and then reconstructs the features to reduce over-segmentation.•Dilation passing network propagates long- and short-range features enabling better understanding of the relation between frames.•Temporal reconstruction network uses a convolutional encoder-decoder to capture local context for temporal consistency among frames.•Our model achieves meaningful results over the state-of-the-art models on three challenging datasets.