Video anomaly understanding (VAU) is critical for real-world scenarios. Recent advances in Video Large Language Models (Video-LLMs) enhance the ability of VAU models to describe and interpret anomalies. However, progress in anomaly localization is still limited by two key issues. First, most existing video anomaly datasets only annotate segments that are clearly inconsistent with the context, often omitting subsequent segments that are semantically part of the same abnormal event. Second, the field lacks systematic evaluation protocols. To bridge these gaps, we introduce VALU, a new benchmark that explicitly defines anomalies across five semantic levels and provides comprehensive temporal boundaries and detailed textual descriptions for each. Based on these annotations, we design three evaluation tasks that comprehensively assess models’ capabilities across different dimensions, including temporal grounding, anomaly localization, and anomaly detail discrimination. Evaluation results reveal persistent challenges in current models’ capabilities on VAU. We further analyze and discuss these findings, and hope that both VALU and insights will advance research in VAU and the development of Video-LLMs. Our benchmark will be publicly available.

Video Grounding (VG) aims to identify the moment-of-interest in unedited videos according to given language queries. Zero-shot methods seek to achieve grounding results without labeled data, thereby demonstrating robust generalization capabilities and extensive applicability. However, existing zero shot approaches face limitations when matching queries with videos that contain rich information, primarily due to the intricate structure and semantics of the queries. Firstly, these methods tend to overlook the inherent temporal structure of events described by queries, which includes distinct phases: the initial, climax, and decay stages. By treating all phases of an event uniformly, critical video segments may be inadvertently omitted. Secondly, for queries exhibiting similar semantics, ambiguity can occur during the grounding process, compounded by the lack of reliable verification mechanisms to assess or correct these results. To tackle these challenges, we propose a two-stage method: Exploiting Prior Tacit Knowledge to Enhance Alignment and Verification. Specifically, in the first stage, we introduce a Temporal Structure-Informed Event Proposal Generation (TSPG) module,which capitalizes on the temporal structure of events to effectively filter video candidate segments that encapsulate critical phases of the events. In the second stage, we present a Temporal Consistency Verification and Recalibration (TCVR) module, designed to rigorously examine and refine the grounding results in accordance with the established prior semantic temporal order. Extensive ablation on two datasets demonstrates the superiority of our method.

Transformer-based pretrained language models (PLMs) have achieved great success in modern NLP. An important advantage of PLMs is good out-of-distribution (OOD) robustness. Recently, diffusion models have attracted a lot of work to apply diffusion to PLMs. It remains under-explored how diffusion influences PLMs on OOD data. The core of diffusion models is a forward diffusion process which gradually applies Gaussian noise to inputs, and a reverse denoising process which removes noise. The noised input reconstruction is a fundamental ability of diffusion models. We directly analyze OOD robustness by measuring the reconstruction loss, including testing the abilities to reconstruct OOD data, and to detect OOD samples. Experiments are conducted by analyzing different training parameters and data statistical features on eight datasets. It shows that finetuning PLMs with diffusion degrades the reconstruction ability on OOD data. The comparison also shows that diffusion models can effectively detect OOD samples, achieving state-of-the-art performance in most of the datasets with an absolute accuracy improvement up to 18%. These results indicate that diffusion reduces OOD robustness of PLMs.

A hybrid network with deep integration of communication and computing resources is the development trend of future 6G wireless networks. Achieving energy-efficiency is considered one of the most important goals for collaborative resource allocation in hybrid network optimization. In this paper, we first model communication and computing resources, and optimize the energy consumption of base stations while ensuring quality of user experience. Then, the performance of the patch-on AI introduction mode under 5G and the 6G native AI were analyzed and verified. Finally, data augmentation is used to optimize the generalization and adaptability of the native AI model. Compared with no data augmentation, our solution improves the reward by 2.38% while data missing,improves the reward by 4.16% while data expansion and improves the reward by 10.04% while data interference.

RGB and depth modalities provide complementary information, which can be effectively utilized to improve the performance of hand detection in the wild. Most existing fusion-based methods model the channel-wise or spatial-wise cross-modal correlation to exploit the complementary RGB-D information, in which the modeling operations are shared across all input samples. However, the input images show various modes due to the high diversity of scenes in the wild. This inter-sample variance cannot be effectively perceived by static modeling operations shared across all samples. To address this problem, we propose a Sample-Adapt Fusion Network (SAFNet) with Channel Dynamic Refinement Module (CDRM) and Spatial Dynamic Aggregation Module (SDAM) to adaptively model the channel-wise and spatial-wise cross-modal correlation. Specifically, we propose a Multi-kernel Attention Module (MAM) to individually generate attention maps for each input sample by applying learnable weighting operations to multiple convolutional kernels. Our method outperforms state-of-the-art methods on CUG Hand dataset.

Web-based AR technology has broadened human-computer interaction scenes from traditional mechanical devices and flat screens to the real world, resulting in unconstrained environmental challenges such as complex backgrounds, extreme illumination, depth range differences, and hand-object interaction. The previous hand detection and 3D hand pose estimation methods are usually based on single modality such as RGB or depth data, which are not available in some scenarios in unconstrained environments due to the differences between the two modalities. To address this problem, we propose a multimodal fusion approach, named Scene-Adapt Fusion (SA-Fusion), which can fully utilize the complementarity of RGB and depth modalities in web-based HCI tasks. SA-Fusion can be applied in existing hand detection and 3D hand pose estimation frameworks to boost their performance, and can be further integrated into the prototyping AR system to construct a web-based interactive AR application for unconstrained environments. To evaluate the proposed multimodal fusion method, we conduct two user studies on CUG Hand and DexYCB dataset, to demonstrate its effectiveness in terms of accurately detecting hand and estimating 3D hand pose in unconstrained environments and hand-object interaction.

Video anomaly detection aims to automatically detect abnormal objects or behaviors. Most existing methods tackle the problem by minimizing the reconstruction errors stemming from the lack of anomalous data, which leads to poor interpretability and robustness. Focus on the context-dependent nature of anomaly detection, a robust unsupervised Video Anomaly Detection framework based on Knowledge and Frame Prediction is proposed, called VAD-KFP. Prior knowledge which contains the context of anomaly is introduced into the multi-path frame prediction network through multi-layer Graph Convolutional Networks. By integrating the prior knowledge to accurately define anomalies, VAD-KFP is robust to different scenarios and is able to recognize the type of anomaly. An extensive range of experiments have been conducted on three benchmarks, the results of which indicate that our method outperforms strong baselines. Specifically, VAD-KFP obtains an AUROC score of 91.6% for the Avenue dataset.

3D hand reconstruction from RGB image has attracted a lot of attention due to its crucial role in human-computer interaction. Nevertheless, it is still challenging to perform 3D hand reconstruction under conditions of hand-object interaction due to severe mutual occlusion. Previous methods usually adopt fixed convolution kernel to extract features. We argue that simply sharing the static filter for all regions is impertinent, given that the occlusion degree varies across different regions, resulting in inconsistent visual representations. To address this issue, we proposed Region-aware Dynamic Filtering Network (RDFNet), which dynamically generates convolution kernels based on the features of different regions, thereby adaptively extracting region-related information. Furthermore, we introduce a dynamic receptive field selection mechanism to determine the most appropriate scale for the convolution kernel. For the severely occluded regions, larger receptive field is needed to capture semantic-related features, while the visible regions are mainly concerned with their own local pattern to accumulate spatial-related features and avoid the interference of irrelevant information. Our proposed RDFNet outperforms state-of-the-art methods by a large margin on several challenging hand-object interaction datasets.

Existing weakly-supervised temporal sentence grounding methods typically regard query reconstruction as the pretext task in place of the absent temporal supervision. However, their approaches suffer from two flaws, i.e. insignificant reconstruction and discrepancy in alignment. Insignificant reconstruction indicates the randomly masked words may not be discriminative enough to distinguish the target event from unrelated events in the video. Discrepancy in alignment indicates the incorrect partial alignment built by query reconstruction task. The flaws undermine the reliability of current reconstruction-based methods. To this end, we propose a novel Self-improving Query ReconstrucTion (SQRT) framework for weakly-supervised temporal sentence grounding. To deal with insignificant reconstruction, we devise a key words mining strategy to determine the important words for language grounding. To attain better moment-query alignment, we introduce inter-sample contrast to tackle the partial alignment built by query reconstruction. The self-improving framework utilizes query reconstruction for language grounding and alleviates the discrepancy in alignment, thus turning on the right track. Experiments on two popular datasets show that SQRT achieves state-of-the-art performance on Charades-STA and comparable performance to the state-of-the-art on ActivityNet Captions.