Point-supervised temporal action localization (P-TAL) aims to localize actions in untrimmed videos with only one action-level timestamp annotation. Owing to label sparsity in P-TAL, most existing methods locate action by boundary-free considering classification, which results in incomplete action localization. In this paper, we propose a novel point-to-boundary hierarchical learning framework that generates dense and reliable pseudo-action proposals to provide complete action annotation containing boundaries for a multi-granularity localization model with single-stage inference. Concretely, we first achieve action proposal boundary enhancement which takes the annotated points as corresponding action centers to search for the boundary-accurate action proposals by fitting a Gaussian prior according to Class Activation Sequences. Furthermore, to more noise-resiliently learn boundary information of actions from the obtained proposals, we introduce a hierarchical boundary regression algorithm and a multi-level attention loss function. Extensive experimental results on three challenging benchmarks demonstrate the state-of-the-art performance and the generalization of the proposed framework. Notably, our method even shows faster single-stage inference with better performance compared to the existing methods.

Referring Remote Sensing Image Segmentation (RRSIS) is a challenging task that advances scene understanding in the field of remote sensing image processing. Existing methods predominantly focus on enhancing visual features with auxiliary text inputs, while overlooking mutual cross-modal interactions. In this work, we propose a novel Prompt-Guided Multi-Scale Cross-Modal Fusion Network (PMCFNet) to achieve fine-grained semantic alignment between visual and textual features. The PMCFNet primarily consists of a Dual-Parallel Interaction Module (DPIM) and Cross-layer Multi-scale Fusion Module (CMFM). The DPIM establishes a parallel architecture to facilitate bi-directional information flow, enabling the exploration of complementary information of multi-modalities. The CMFM combines visual and textual features across multiple layers, enhancing the understanding of both global and local features. Additionally, we introduce a prompt-guided learning strategy to further enrich textual representations by embedding location and target-specific knowledge. This strategy is particularly critical for identifying multi-scale objects from high-resolution remote sensing scenarios, significantly improving the model’s discriminative capabilities. Experiments on the RefSegRS and RRSIS-D benchmarks demonstrate that our method achieves state-of-the-art performance, i.e., yielding improvements of 1.91 https://github.com/woaiqianzhihe/PMDF

Temporal action segmentation (TAS) aims to classify the action category of each frame in untrimmed videos, with RGB videos or skeleton sequences as common inputs. Most existing methods follow a two-stage pipeline of feature extraction and temporal modeling. However, we observe two key limitations in their spatio-temporal modeling: (i) Existing temporal modeling modules conduct frame-level and action-level interactions at a single fixed temporal resolution, which over-smooths temporal features and blurs action boundaries; (ii) Skeleton-based methods generally adopt temporal modeling modules originally designed for RGB video data, causing a misalignment between extracted features and temporal modeling modules. To address these issues, we propose a novel Interaction-based framework for Action segmentation (InterAct). Firstly, we introduce multi-scale frame-action interaction (MFAI) to facilitate frame-action interactions across varying temporal scales. This enhances the model’s ability to capture complex temporal dynamics, producing more expressive temporal representations and alleviating the over-smoothing issue. Meanwhile, recognizing the complementary nature of different spatial modalities, we further propose decoupled spatial modality interaction (DSMI). It decouples the modeling of spatial modalities and applies a deep fusion strategy to interactively integrate multi-scale spatial features. This results in more discriminative representations that better support temporal modeling. Extensive experiments on seven large-scale benchmarks show that InterAct significantly outperforms state-of-the-art methods on both RGB-based and skeleton-based TAS. Further evaluations on multi-person interactions and simulated occlusions demonstrate improved robustness under imperfect skeleton observations. The code is available at https://github.com/gdxxu/InterAct-main.

Reinforcement Learning (RL) enables robust and adaptive locomotion in legged and wheeled-legged robots. A common approach is the Teacher-Student (TS) paradigm, in which a teacher policy with privileged information supervises a proprioceptive student. While the TS paradigm has proven effective on legged robots, we encounter two critical issues when applying it to wheeled-legged robots. One issue is multimodal confusion, where teacher actions become multimodal under the student proprioceptive observations, resulting in the student generating averaged action modes. The other is low imitability of teacher actions, as the teacher overlooks their reproducibility by the student. To address these issues, we propose Teaching to Individual Needs (TIN), a bidirectional TS framework. To mitigate multimodal confusion within the student policy, we design a Highest-Weight Component Mixture Density Network (HWC-MDN). By utilizing HWC-MDN, TIN student can explicitly model multimodal action distributions and outputs the highest-weight component. To improve imitability, we propose an Imitation-Aware Reward (IAR) that encourages the teacher to generate more reproducible actions by the student. Simulation experiments show that TIN significantly improves both training efficiency and traversability. Real-world tests illustrate that TIN enables the wheeled-legged robot MagicDog-W to traverse 45 cm obstacles and ascend 45° slopes.

In unknown environments, navigating a robot by a given image to a specific location or instance is critical and challenging. The existing end-to-end approaches require simultaneous implicit learning of multiple subtasks, and modular approaches depend on metric information. Both approaches face high computational demands, often leading to difficulties in real-time updates and limited generalization, making them challenging to implement on resource-constrained devices. To address these challenges, we propose Dual Graph Navigation (DGN), a knowledge-driven, lightweight image instance navigation framework. DGN builds an External Knowledge Graph (EKG) from small-scale datasets to capture prior object correlations, efficiently guiding target exploration. During exploration, DGN builds an Internal Knowledge Graph (IKG) using an instance-aware module, which records explored objects based on reachability relationships rather than precise metric information. The IKG dynamically updates the EKG, enhancing the robot's adaptability to the current environment. Together, they realize topological perception and reduce computational overhead. Furthermore, unlike approaches characterized by over-dependence between components, DGN employs a plug-and-play modular design that allows independent training and flexible replacement of functional modules, effectively enhancing generalization performance while reducing training and deployment costs. Experiments illustrate that DGN generalizes well in different simulation environments (AI2-THOR, Habitat), achieving state-of-the-art performance on the ProcTHOR-10K dataset. It is compatible with three distinct real-world robot platforms, including edge computing devices without CUDA support. It exhibits a decision-making speed of 3.8 to 5.5 times over baseline methods. Further details can be found on the project page: https://dogplanningloyo.github.io/DGN/.

Temporal distribution shifts occur not only in low-dimensional time-series data but also in high-dimensional data like videos. This phenomenon leads to significant performance degeneration in video understanding methods such as streaming temporal action segmentation. To address this issue, we propose a flexible streaming temporal action segmentation model with diffusion models (FSTAS-DM). By utilizing streaming video clips with varying feature distributions as control conditions, our model can adapt to the shifts and inconsistency of the distribution between the training and testing domains. Additionally, we have introduced a multistage conditional control training strategy (MSCC), which enhances the temporal generalization ability of the model. Our method demonstrates commendable performance on datasets like GTEA, 50Salads, and Breakfast.

Recently, deep learning-based remote sensing image super-resolution (RSISR) techniques have achieved significant progress, but challenges remain in preserving critical edge details essential for high-quality image reconstruction. These details are crucial for tasks such as object recognition, change detection, and accurate analysis in remote sensing imagery. Furthermore, existing RSISR methods typically require substantial computational resources, making them unsuitable for resource-constrained edge devices. To address these challenges, we propose a novel edge-guided super-resolution network (EGSRN). The network employs an edge extraction module (Edge Net) to explicitly extract edge information from low-resolution (LR) images, combined with multilayer feature extraction modules (FEMs) and an edge information fusion (EIF) mechanism to progressively integrate edge and image features. This design enables precise recovery of edge details, significantly enhancing the overall visual quality of the reconstructed images. Edge-aware processing enhances visual fidelity while also improving the accuracy of downstream tasks, such as classification, object detection, and change analysis. Furthermore, the network incorporates lightweight designs such as depthwise separable convolutions and channel shuffling to effectively reduce computational demands. Comprehensive experiments were conducted on two remote sensing datasets, and the model's parameter count and floating-point operations (FLOPs) were evaluated. Results demonstrate that the proposed method achieves an excellent balance between performance and model complexity, delivering superior super-resolution reconstruction quality while maintaining low computational costs, making it well-suited for resource-limited real-world applications.

Recently, deep learning-based remote sensing image super-resolution (RSISR) has achieved remarkable progress. However, conventional super-resolution (SR) methods usually assume a fixed and known degradation process (e.g., bicubic downsampling), which often leads to significant performance degradation when applied to real-world data with diverse and unknown degradations. To overcome this limitation, we propose DCR-SRNet, a novel degradation-contrastive and wavelet-guided network for blind RSISR. The proposed network incorporates three key innovations: First, we design a contrastive degradation representation learning strategy that disentangles degradation priors from scene semantics by pulling together representations of identical degradations across different scenes while pushing apart those of different degradations within the same scene. Second, we introduce a wavelet-guided patch-wise weighted loss module, which employs wavelet decomposition and patch-level discrimination scores to adaptively reweight the pixel-wise loss, thereby enhancing the recovery of edge and texture details. Third, we design an adaptive modulation block (AMB) that injects degradation priors into the reconstruction process through feature- and channel-wise modulation, enabling robust adaptation to diverse degradations. Extensive experiments on three benchmark remote sensing datasets demonstrate that DCR-SRNet significantly outperforms state-of-the-art methods, particularly in preserving structural and textural details.

Existing multimodal large language models (MLLMs) face significant challenges in Referring Video Object Segmentation(RVOS). We identify three critical challenges: (C1) insufficient quantitative representation of textual numerical data, (C2) repetitive and degraded response templates for spatiotemporal referencing, and (C3) loss of visual information in video sampling queries lacking textual guidance. To address these, we propose a novel framework, Dynamic Time Object Sensing (DTOS), specifically designed for RVOS. To tackle (C1) and (C2), we introduce specialized tokens to construct multi-answer response templates, enabling regression of event boundaries and target localization. This approach improves the accuracy of numerical regression while mitigating the issue of repetitive degradation. To address (C3), we propose a Text-guided Clip Sampler (TCS) that selects video clips aligned with user instructions, preventing visual information loss and ensuring consistent temporal resolution. TCS is also applicable to Moment Retrieval tasks, with enhanced multimodal input sequences preserving spatial details and maximizing temporal resolution. DTOS demonstrates exceptional capability in flexibly localizing multiple spatiotemporal targets based on userprovided textual instructions. Extensive experiments validate the effectiveness of our approach, with DTOS achieving state-of-the-art performance in J&F scores: an improvement of +4.36 on MeViS, +4.48 on Ref-DAVIS17, and +3.02 on Ref-YT-VOS. Additionally, our TCS demonstrates exceptional performance in Moment Retrieval. The code is available at https://github.com/Maulog/OPEN-DTOS-LMM.

The streaming temporal action segmentation (STAS) task, a supplementary task of temporal action segmentation (TAS), has not received adequate attention in the field of video understanding. Existing TAS methods are constrained to offline scenarios due to their heavy reliance on multimodal features and complete contextual information. The STAS task requires the model to classify each frame of the entire untrimmed video sequence clip by clip in time, thereby extending the applicability of TAS methods to online scenarios. However, directly applying existing TAS methods to SATS tasks results in significantly poor segmentation outcomes. In this article, we thoroughly analyze the fundamental differences between STAS tasks and TAS tasks, attributing the severe performance degradation when transferring models to model bias and optimization dilemmas. We introduce an end-to-end streaming video TAS model with reinforcement learning (SVTAS-RL). The end-to-end modeling method mitigates the modeling bias introduced by the change in task nature and enhances the feasibility of online solutions. Reinforcement learning (RL) is utilized to alleviate the optimization dilemma. Through extensive experiments, the SVTAS-RL model significantly outperforms existing STAS models and achieves competitive performance to the state-of-the-art (SOTA) TAS model on multiple datasets under the same evaluation criteria, demonstrating notable advantages on the ultralong video dataset EGTEA. Our code is publicly available at https://github.com/Thinksky5124/SVTAS.

Recently, deep learning-based remote sensing image super-resolution (RSISR) methods have achieved remarkable progress. However, effectively preserving high-frequency details remains a significant challenge, as these features are critical for downstream tasks such as object detection, change analysis, and scene classification. Moreover, relying solely on the information contained in low-resolution (LR) images often results in the loss of structural details, thereby degrading reconstruction quality. To address these issues, we propose a novel wavelet-guided and feature-aware super-resolution network (WFA-SRNet). The proposed network adopts a dual-branch architecture, consisting of a feature extraction block (FEB) and a high-frequency extraction (HFE), to collaboratively model semantic structures and fine-grained textures. Specifically, FEB integrates a shift-window cross attention (SWCA) mechanism and a dictionary-based similarity matching strategy to capture nonlocal self-similarities, while the HFE branch incorporates a wavelet-domain high-frequency enhancement module (WD-HFE), which explicitly decomposes and reconstructs frequency components via discrete wavelet transform (DWT) and inverse DWT (IDWT) to enhance edge and texture recovery. Furthermore, a fusion attention (FA) module is designed to guide the integration of multisource features from both semantic and high-frequency pathways. Extensive experiments on multiple benchmark remote sensing datasets demonstrate that WFA-SRNet achieves superior reconstruction performance, particularly in restoring structural and textural details. Additionally, the proposed method significantly improves the accuracy of downstream classification tasks, showing strong potential for practical RSISR applications.

Commentators can provide rich information and analysis to bring good experience to the audience, leading to great application prospects for AI commentators. In this paper, we propose Fine-grained Video Action commentary (FVAC) as a new challenging task that uses long natural language to comment the action in the input video like a real commentator. In addition to describing the action, FVAC focuses on providing detailed evaluation, background knowledge, and technical instructions. Due to the high difficulty to obtain detailed semantic information in the video and the lack of background knowledge, existing researches are based only on action evaluation and description. Fortunately, the great achievements of AI agent in general artificial intelligence give us new solutions. To solve the task, we propose a strong AI agent called the Commentary Master framework to generate fine-grained commentary. In the agent, we propose Discriminator Net (DNet) to extract detailed semantic information and Balance Tree Retrieval-Augmented Generation (BTRAG) to provide rich background knowledge and examples. Additionally, to address the difficulty of evaluating the commentary content with rich styles, we propose the Commentary Quality Level (CQL) metric. Compared to base LVLM models, our experiment shows that the Commentary Master can generate high-quality fine-grained commentary.

Manifold learning now plays an important role in machine learning and many relevant applications. In spite of the superior performance of manifold learning techniques in dealing with nonlinear data distribution, their performance would drop when facing the problem of data sparsity. It is hard to obtain satisfactory embeddings when sparsely sampled high-dimensional data are mapped into the observation space. To address this issue, in this article, we propose hierarchical neighbors embedding (HNE), which enhances the local connections through hierarchical combination of neighbors. And three different HNE-based implementations are derived by further analyzing the topological connection and reconstruction performance. The experimental results on both the synthetic and real-world datasets illustrate that our HNE-based methods could obtain more faithful embeddings with better topological and geometrical properties. From the view of embedding quality, HNE develops the outstanding advantages in dealing with data of general distributions. Furthermore, comparing with other state-of-the-art manifold learning methods, HNE shows its superiority in dealing with sparsely sampled data and weak-connected manifolds.

For RGB-based temporal action segmentation (TAS), excellent methods that capture frame-level features have achieved remarkable performance. However, for motion-centered TAS, it is still challenging for existing methods that ignore the extraction of spatial features of joints. In addition, inaccurate action boundaries caused by the frames of similar motion destroy the integrity of the action segments. To alleviate the issues, an end-to-end Involving Distinguished Temporal Graph Convolutional Networks called IDT-GCN is proposed. First, we construct an enhanced spatial graph structure that adaptively captures the similar and differential dependencies between joints in a single topology through learning two independent correlation modeling functions. Then, the proposed Involving Distinguished Graph Convolutional (ID-GC) models the spatial correlations of different actions in a video by using multiple enhanced topologies on the corresponding channels. Furthermore, we design a generic modeling temporal action regression network, termed Temporal Segment Regression (TSR), to extract segmented encoding features and action boundary representations by modeling action sequences. Combining them with label smoothing modules, we develop powerful spatial-temporal graph convolutional networks (IDT-GCN) for fine-grained TAS, which notably outperforms state-of-the-art methods on the MCFS-22 and MCFS-130 datasets. Adding TSR to TCN-based baseline methods achieves competitive performance compared with the state-of-the-art transformer-based methods on RGB-based datasets, i.e., Breakfast and 50Salads. Further experimental results on the action recognition task verify the superiority of the enhanced spatial graph structure over the previous graph convolutional networks.

Human Action Quality Assessment (AQA) is a prominent area of research in human action analysis. Current mainstream methods only consider the RGB modality which results in limited feature representation and insufficient performance due to the complexity of the AQA task. In this paper, we propose a simple and modular framework called the Two-Modality Assessment Framework (2M-AF), which comprises a skeleton stream, an RGB stream and a regression module. For the skeleton stream, we develop the Self-supervised Mask Encoder Graph Convolution Network (SME-GCN) to achieve representation learning, and further implement score assessment. Additionally, we propose a Preference Fusion Module (PFM) to fuse features, which can effectively avoid the disadvantages of different modalities. Our experimental results demonstrate the superiority of the proposed 2M-AF over current state-of-the-art methods on three publicly available datasets: AQA-7, UNLV-Diving, and MMFS-63. (1 2)

MicroRNAs (miRNAs) play important roles in the occurrence and development of diseases. However, it is still challenging to identify the effective miRNA biomarkers for improving the disease diagnosis and prognosis. In this study, we proposed the miRNA data analysis method based on multi-view miRNA networks and reinforcement learning, miRMarker, to define the potential miRNA disease biomarkers. miRMarker constructs the cooperative regulation network and functional similarity network based on the expression data and known miRNA-disease relations, respectively. The cooperative regulation of miRNAs was evaluated by measuring the changes of relative expression. Natural language processing was introduced for calculating the miRNA functional similarity. Then, miRMarker integrates the multi-view miRNA networks and defines the informative miRNA modules through a reinforcement learning strategy. We compared miRMarker with eight efficient data analysis methods on nine transcriptomics datasets to show its superiority in disease sample discrimination. The comparison results suggested that miRMarker outperformed other data analysis methods in receiver operating characteristic analysis. Furthermore, the defined miRNA modules of miRMarker on colorectal cancer data not only show the excellent performance of cancer sample discrimination but also play significant roles in the cancer-related pathway disturbances. The experimental results indicate that miRMarker can build the robust miRNA interaction network by integrating the multi-view networks. Besides, exploring the miRNA interaction network using reinforcement learning favors defining the important miRNA modules. In summary, miRMarker can be a hopeful tool in biomarker identification for human diseases.

The underwater images often suffer from color deviations and blurred details. To address these issues, many methods employ networks with an encoder/decoder structure to enhance the images. However, the direct skip connection overlooks the differences between pre- and post-features, and deep network learning introduces information loss. This paper presents an underwater image enhancement network that focuses on pre-post differences. The network utilizes a multi-scale input and output framework to facilitate the underwater image enhancement process. A novel cross-wise transformer module (CTM) is introduced to guide the interactive learning of features from different periods, thereby enhancing the emphasis on detail-degraded regions. To compensate for the information loss within the deep network, a feature supplement module (FSM) is devised for each learning stage. FSM merges the multi-scale input features, effectively enhancing the visibility of underwater images. Experimental results across several datasets demonstrate that the integrated modules yield significant enhancements in network performance. The proposed network exhibits outstanding performance in both visual comparisons and quantitative metrics. Furthermore, the network also exhibits good adaptability in additional visual tasks without the need for parameter tuning. Code and models are released in https://github.com/WindySprint/CTM.

Fine-grained Temporal Action Segmentation (TAS) poses greater challenges compared to general temporal action segmentation. Fine-grained TAS requires distinguishing subtle differences among similar actions and accurately modeling along with spatio-temporal attributes. However, previous methods have largely ignored the exploration of spatio-temporal properties, leading to bad performance on fine-grained datasets. In this paper, we propose a novel Decoupling Spatio-Temporal Network (DSTN) that includes the Action Segmentation Expert (ASE) and the Semantic Information Decision Map (SIDM). The DSTN aims to obtain independent action semantics by decoupling spatio-temporal features and attributes. The ASE consists of Spatial-ASE and Temporal-ASE for frame-by-frame action segmentation, while SIDM is used to align spatio-temporal attribute action labels to ensure the rationality of the results. By leveraging pre-existing spatio-temporal attribute actions, DSTN enables zero-shot TAS and the identification of new actions. Furthermore, recognizing the limitations of current datasets, we construct the FineSkating dataset specifically for Fine-grained TAS. Our model outperforms or competes with the state-of-the-art methods on three challenging datasets.

Multi-target tracking has made significant progress in recent years as a key player in the field of computer vision, but remains a challenging problem due to target similarity and complexity. In recent years, deep learning has pushed the field forward, and detection-based tracking methods use an end-to-end strategy that unifies target detection and trajectory modeling in a neural network framework, however, little use is made of appearance information. In this paper, we propose a Siamese structure-based approach that introduces an appearance search branch, aiming to enhance the system's ability to model the utilization of target appearance information. The method is validated on the basis of the FairMOT model, which generates a heat map reflecting the results of the target appearance search by means of feature vectors with multiple time dimensions and the Siamese module. The results of the detection branch and the appearance search branch are fused to form a final multi-target tracking system through post-processing and matching. Experiments demonstrate that the method achieves significant performance improvements over existing methods. This research provides a new perspective on the multi-target tracking problem, enhances the modeling and use of target appearance information through the appearance search branch, and provides an effective tool for system performance improvement in complex scenarios.

The goal of unsupervised action segmentation (UAS) is to classify video frames into predefined action classes, which can be considered as a clustering or boundary detection problem. Previous research utilizing bottom-up agglomerative hierarchical clustering methods suffers from over-segmentation or under-segmentation. To address these problems, we propose the Two-step Temporal Divisive Clustering (TTDC) with two components. The first step of TTDC is top-down Temporal Divisive Clustering (TDC), which captures global contexts by comparing the intra-class variances of different classes, and captures local contexts through boundary detection. The second step is the Self-supervised Soft Boundary Regression Network (SS-BRN). SS-BRN is trained by soft pseudo-labels from TDC to refine the boundaries of clusters. In addition, to alleviate the issue of low confidence in pseudo-labels, we use a loss function with soft pseudo-labels. Our empirical evaluations on three benchmarks including 50Salads, Breakfast, and MPII Cooking 2 dataset demonstrate that TTDC outperforms the state-of-the-art methods.