Interaction Makes Better Segmentation: an Interaction-Based Framework for Temporal Action Segmentation

Temporal action segmentation (TAS) aims to classify the action category of each frame in untrimmed videos, with RGB videos or skeleton sequences as common inputs. Most existing methods follow a two-stage pipeline of feature extraction and temporal modeling. However, we observe two key limitations in their spatio-temporal modeling: (i) Existing temporal modeling modules conduct frame-level and action-level interactions at a single fixed temporal resolution, which over-smooths temporal features and blurs action boundaries; (ii) Skeleton-based methods generally adopt temporal modeling modules originally designed for RGB video data, causing a misalignment between extracted features and temporal modeling modules. To address these issues, we propose a novel Interaction-based framework for Action segmentation (InterAct). Firstly, we introduce multi-scale frame-action interaction (MFAI) to facilitate frame-action interactions across varying temporal scales. This enhances the model’s ability to capture complex temporal dynamics, producing more expressive temporal representations and alleviating the over-smoothing issue. Meanwhile, recognizing the complementary nature of different spatial modalities, we further propose decoupled spatial modality interaction (DSMI). It decouples the modeling of spatial modalities and applies a deep fusion strategy to interactively integrate multi-scale spatial features. This results in more discriminative representations that better support temporal modeling. Extensive experiments on seven large-scale benchmarks show that InterAct significantly outperforms state-of-the-art methods on both RGB-based and skeleton-based TAS. Further evaluations on multi-person interactions and simulated occlusions demonstrate improved robustness under imperfect skeleton observations. The code is available at https://github.com/gdxxu/InterAct-main.