Video moment retrieval (MR) and highlight detection (HD) are two fundamental tasks in multimodal video understanding. They focus on moment localization and key clip selection, respectively, and are widely applied in content recommendation and intelligent retrieval. Although differing in objectives and outputs, both tasks involve vision-language alignment and complex temporal structure modeling, thus possessing natural synergy. However, existing joint modeling approaches often fail to fully exploit structural information in videos, leading to degraded performance in scenarios with ambiguous semantic boundaries or frequent behavioral transitions. To overcome these challenges, we propose a structure-aware joint modeling framework, SE-DETR (Structure-Enhanced DETR), to enhance structural representation and task interaction between MR and HD via Saliency-Guided Structural Retrieval and Boundary-Driven Highlight Re-estimation. SE-DETR reconstructs temporal structure features through a structure-guided module that incorporates slot-based aggregation and boundary response prediction, providing a unified structural prior for downstream tasks. Moreover, a task interaction mechanism is developed, unifying a saliency-guided structural retrieval path with a boundary-driven highlight re-estimation path, thereby enhancing the model’s adaptability to abrupt semantic transitions and improving the precision of target localization. Additionally, a boundary supervision loss based on semantic variation rate is introduced to refine boundary sensitivity without requiring additional annotations, enhancing the model’s ability to represent vague behavior boundaries. Under identical features and evaluation protocols, SE-DETR achieves strong overall performance on QVHighlights, Charades-STA, and TVSum, attaining the highest overall mean AP across these benchmarks compared with recent methods, without requiring extra annotations.

Video Moment Retrieval (VMR) serves as a fundamental task in video understanding, bridging vision and language by localizing the most relevant temporal segments in untrimmed videos according to a textual query. However, existing approaches excel at fine-grained alignment but often fail to capture global temporal context effectively, particularly in long-form videos. To address this challenge, we propose Hybrid Mamba Network (HM-Net), a two-level fusion architecture which unifying the strengths of attention and sequence modeling. Especially, its core lies in the Hybrid Modulated Bi-Mamba (HMB) Block, which integrates the powerful temporal modeling capability of Mamba into the VMR framework to achieve effective long-range temporal reasoning. Extensive experiments on the challenging TACoS and QVHighlights benchmarks show that HM-Net consistently outperforms existing approaches, achieving 3.84% improvement in R1@0.5 (TACoS) and 1.65% in mAP (QVHighlights), demonstrating notable gains in localization accuracy, particularly on long-form videos.

Realistic image super-resolution (RISR) has been a challenging research topic in image restoration, aiming to address complex degradation. However, existing methods often struggle to handle various unknown degradation factors presented in low-quality images, limiting their effectiveness to simplify degradation models. This gap between current RISR methods and real-world scenarios hinders their ability to generate realistic details. In this paper, we propose a novel realistic image super-resolution method based on stable diffusion to enhance degradation perception and detail generation. The framework comprises a Degradation-aware Module (DAM), a Detail-enhanced Module (DEM), and a General Restoration Module (GRM). DAM adjusts the weights of deep features in different channels using the Residual Composite Attention Network (RCAN) to remove perceived degradation, producing a smooth image containing only essential information. DEM employs a feature enhancement structure from coarse to fine to transform low-dimensional feature data obtained by downsampling the image data into corresponding high-resolution image data. GRM utilizes a pre-trained stable diffusion model based on the inverse diffusion process, and we add Advanced Nets in the diffusion stage to provide additional semantic information for noise restoration. Experimental results demonstrate the superiority of our method over existing state-of-the-art methods on both synthetic and real-world datasets.

Objective Image colorization refers to the process of predicting plausible colors for each pixel in a grayscale image,with the goal not necessarily being an exact restoration of the original color.Given that the same object can often be assigned different colors,colorization inherently exhibits multimodal characteristics.This complexity presents challenges for researchers and sustained interest in the field.As an important direction in computer vision,image colorization has gained considerable attention,particularly with the advancements in deep learning.Traditional colorization methods often rely on user input,such as scribble-based or reference image-based techniques,which,while effective,are hindered by limitations such as the inability to handle batch processing and extended processing times.To address these issues and reduce manual intervention,deep learning techniques have propelled the development of automatic colorization.Fully auto-matic methods,which eliminate the need for user interaction,can efficiently colorize images;however,challenges includ-ing the accuracy of color restoration and the preservation of image details remain.Deep learning approaches,particularly those based on convolutional neural networks and generative adversarial networks,have demonstrated improvements in col-orization performance but continue to face issues such as insufficient semantic understanding and blurred details.Recently,Transformer models have shown promise in image colorization tasks by leveraging their ability to capture long-range dependencies,further enhancing results.However,existing methods still struggle with challenges such as color bleeding and loss of detail clarity,especially in highly detailed images.Achieving fully automated,natural,and plausible colorization remains an ongoing research challenge.Method In this study,we propose an end-to-end grayscale image color-ization method utilizing encoder-decoder architecture for fully automatic colorization.Given an input grayscale image,the network predicts the chrominance channels in the CIELAB color space to generate a colorized image.The encoder employs ConvNeXt to leverage its multiscale semantic representation capabilities,effectively extracting high-level semantic features from the grayscale image.Multiscale feature maps are passed from the encoder to the decoder through convolutional connec-tion layers,progressively restoring the image's spatial resolution.These feature maps are then fed into the color prediction network,where a pixel enhancement block(PEB)refines the color predictions.The PEB is designed to focus on and enhance specific regions of the image,improving color matching accuracy by utilizing convolutional layers and pooling operations to generate spatial attention weights.These weights are element-wise multiplied with the original image,enabling spatial enhancement and better capturing important regions for improved color precision.The color query block in the color prediction network employs a Transformer-based approach,incorporating learnable color embedding memories that store sequences of color representations.Through cross-attention and self-attention mechanisms,color embeddings are progressively correlated with image features,reducing dependence on manual priors and improving sensitivity to semantic information.This approach mitigates issues such as color bleeding.Furthermore,to enhance the learning of color informa-tion and latent features from the grayscale image,the feature enhancement block generates attention maps using convolu-tional operations with varying kernel sizes.These maps are fused with the original image through convolutional layers to pro-duce the final output tensor.This methodology ensures effective reconstruction of color and structural information,thereby enhancing the overall performance of the image colorization process.Result In the experiments,the proposed colorization model was trained on the large-scale ImageNet dataset and extensively evaluated across multiple benchmark datasets,including ImageNet(val5k),ImageNet(val50k),COCO-Stuff,ADE20K,and CelebA-HQ.The evaluation metrics included Frechet inception distance(FID),colorfulness score(CF),and peak signal-to-noise ratio(PSNR),which assess the realism,color quality,and reconstruction accuracy of the generated images.The model utilized a pretrained ConvNeXt-L as the encoder,paired with a multiscale decoder,and was optimized using the Adam W optimizer.All experiments were performed on four Tesla A100 GPUs.Comparative results showed that the proposed method remarkably outperformed exist-ing approaches such as DeOldify,Wu et al.,BigColor,ColorFormer,and DDColor,particularly in terms of color richness and realism.Quantitative comparisons across five test datasets further demonstrated that the proposed model consistently achieved the lowest FID scores,indicating superior image quality and strong generalization.Compared with the second-best model,the FID is reduced by 0.2,while the PSNR is improved by 0.13 dB.While previous methods achieved higher CF scores,a higher colorfulness score does not always correlate with better visual quality.Thus,the metric △CF was intro-duced to measure the difference in colorfulness between generated and real images.The proposed method achieved the low-est △CF scores across all datasets,reflecting its ability to generate more natural and realistic colorizations while preserving image diversity.Given the subjective nature of image colorization,a user study was also conducted,showing that over 30％of users preferred the colorization results produced by the proposed method.Additionally,ablation studies confirmed the effectiveness of the model architecture in enhancing colorization performance.Conclusion The proposed colorization model is good at capturing and reproducing the details and color relationships in the image,achieving high-quality colorization results.

Universal image restoration (UIR) aims to accurately restore images with a variety of unknown degradation types and levels. Existing methods, including both learning-based and prior-based approaches, heavily rely on low-quality image features. However, it is challenging to extract degradation information from diverse low-quality images, which limits model performance. Furthermore, UIR necessitates the recovery of images with diverse and complex types of degradation. Inaccurate estimations further decrease restoration performance, resulting in suboptimal recovery outcomes. To enhance UIR performance, a viable approach is to introduce additional priors. The current UIR methods have problems such as poor enhancement effect and low universality. To address this issue, we propose an effective framework based on a diffusion model (DM) for universal image restoration, dubbed ETDiffIR. Inspired by the remarkable performance of text prompts in the field of image generation, we employ text prompts to improve the restoration of degraded images. This framework utilizes a text prompt corresponding to the low-quality image to assist the diffusion model in restoring the image. Specifically, a novel text–image fusion block is proposed by combining the CLIP text encoder and the DA-CLIP image controller, which integrates text prompt encoding and degradation type encoding into time step encoding. Moreover, to reduce the computational cost of the denoising UNet in the diffusion model, we develop an efficient restoration U-shaped network (ERUNet) to achieve favorable noise prediction performance via depthwise convolution and pointwise convolution. We evaluate the proposed method on image dehazing, deraining, and denoising tasks. The experimental results indicate the superiority of our proposed algorithm.

The development of 2D-to-3D approaches for 3D monocular single-frame human pose estimation faces challenges related to noisy input and failure to capture long-range joint correlations, leading to unreasonable predictions. To this end, we propose a straightforward, but effective U-shaped network called the mapping-aware U-shaped graph convolutional network (M-UGCN) for single-frame applications. This network applies skeletal pooling/unpooling operations to expand the limited convolutional receptive field. For noisy inputs, as local nodes have direct access to the subtle discrepancies between poses, we define an additional mapping-aware local-enhancement mechanism to focus on local node interactions across multiple scales. We evaluated our proposed method on the benchmark datasets Human3.6M and MPI-INF-3DHP, and the experimental results demonstrated the robustness of the M-UGCN against noisy inputs. Notably, the average error in the proposed method was found to be 4.1% lower when compared to state-of-the-art methods adopting similar multi-scale learning approaches.

Currently, common three-dimensional (3D) human pose estimation algorithms achieve good results in representation learning, but still suffer from poor estimation accuracy and depth ambiguity at the joint points of the human skeleton, and extracting the image context is highly promising for mitigating the depth ambiguity. Therefore, an effective way to estimate human pose from monocular video images using redundant two-dimensional (2D) pose sequence spatio-temporal information is a research challenge. Many existing studies have mostly attempted to capture the spatial as well as temporal relationships of human poses in videos to solve these two problems. However, these works tend to overlook the fact that the lack of remote dependency modeling capabilities and parallelism makes the estimated 3D human poses often less accurate. We have proposed a Recurrent Transformer that can achieve a good balance between its efficiency and model size while still maintaining its effectiveness. This method will decompose the task into three phases: (i) serial cyclic spatial relationship modeling of the pose; (ii) parallel cyclic temporal relationship modeling of the pose; and (iii) summarization of multiple features and synthesis of the final 3D pose after linear regression. With the three main processes mentioned above, the final predicted 3D poses are not only improved compared to traditional methods but also more accurate in terms of precision. After experimental validation, Recurrent Transformer achieves leading results on Human3.6M.

Image outpainting aims at generating new looking-realistic content beyond the original boundaries for a given image patch. Existing image outpainting methods tend to generate images with erroneous structures and unnatural colors when extrapolating the sub-image all-side. To solve this problem, we propose a Transformer-based staged image outpainting network. Specifically, we restructure the encoder-decoder architecture by adding hierarchical cross attention to the connection in each layer. We propose a staged expanding module that splits the extrapolation into vertical and horizontal steps so that the generated images can have consistent contextual information and similar texture. A color harmonization module that adjusts both local and global color information is also presented to make color transitions more natural. Our experiments prove that the proposed method outperforms the advanced methods on multiple datasets.

To solve the problems of texture blurring and structure inconsistency in large area image inpainting. This paper proposes an image inpainting method based on multi-scale feature fusion. We design a multi-scale feature fusion module to expand the receptive field. Besides, we devise attention module to capture information from distant areas in the feature map. In this paper, partial convolution and recursive structure are adopted to repair the boundary of the missing area progressively. This method can continuously strengthen the constraint on the center of the missing region and make the repair results more refined. Experimental results show that, compared with the existing image inpainting methods, this structure improves the performance of image inpainting, and the image quality generated by it is the best.

Due to the underdeveloped scanning technology, some old movie films are scanned in digital format with lower resolution, which does not meet the viewing needs of contemporary viewers. Therefore, it is necessary to super-resolution processing them to improve the image quality. However, some old movies will appear blurred after scanning. In this case, the existing algorithm super-resolution reconstruction results are often not ideal. This paper adds image deblurring pre-processing before the super-resolution processing. First, the old movie is deblurred according to the deblurring generation training model against the network, and then the image is super-resolution processed by the sub-pixel convolution network. The method aims to improve the problem that the repair effect caused by the image blur caused by the old film in the super-resolution reconstruction is not ideal.

In the long history of human civilization, archived film is an indispensable part of it, and using digital method to repair damaged film is also a mainstream trend nowadays. In this paper, we propose a sparse color correspondences based technique to remove fading flicker for old films. Our model, combined with multi frame images to establish a simple correction model, includes three key steps. Firstly, we recover sparse color correspondences in the input frames to build a matrix with many missing entries. Secondly, we present a low-rank matrix factorization approach to estimate the unknown parameters of this model. Finally, we adopt a two-step strategy that divide the estimated parameters into reference frame parameters for color recovery correction and other frame parameters for color consistency correction to remove flicker. Our method combined multi-frames takes continuity of the input sequence into account, and the experimental results show the method can remove fading flicker efficiently.

With the changes of eras, traditional movies may suffer continuous frame damage after digits due to improper preservation. In order to solve this problem, we proposed a new inpainting technique for continuous video sequences based on deep neural networks. We introduced the latest image restoration techniques for inpainting key frames of new scenes. Then we use the deep neural network interpolation algorithm to interpolate the intermediate frames, so that the video can achieve a coherent effect in time. In order to preserve the original information as much as possible, we only replace the damaged areas and preserve most of undamaged areas, and finally perform image blending. We test different types of videos and compare them with other methods. Our method has higher quality video inpainting than existing methods.

The colorization of black and white films was a hot topic in the 1980s. Some black-and-white movies regained their luster through colorization. Although people are controversial about the artistic value of film colorization, it is no doubt that color images can enhance visual effects. Inspired by the recent colorization methods using deep learning, we propose a novel colorization model which combines two Convolutional Neural Networks and uses multi-scale convolution kernels to get better spatial consistency. Most of the current datasets used in the colorization networks are not applicable to colorizing images from Chinese black and white films. The main reason is that the objects in these films are very different from today's. To address this, we extract a large number of images from Chinese color films of the last century as a training dataset. Experiments demonstrate that our model can obtain pretty good results of colorizing images from Chinese black and white films.

In old movies, the common jitter is caused by translation, rotation and zooming. Aiming at the common phenomenon of video jitter, this paper proposes a method of combining Lucas-Kanade sparse optical flow with feature point matching to estimate the global motion parameters. Then, it is applied to the restoration of old film, so as to realize the motion compensation from the jitter frame to the reference frame, so as to achieve the image stabilization effect of the continuous sequence frame of the old movie. Experiment's results show that this algorithm has a good real-time performance, and it can solve the problem of smooth transition between frame and frame effectively.

The vintage films may contain large regions of damaged content which is hard to be restored. To solve this problem, we proposed an video inpainting algorithm depended on the optimization of a global, patch-based functional. Our method is divided into three steps. Firstly, we use color transfer to normalize the average intensity of each frame. Then, we take a non-local patch-based approach to search the nearest neighbor of the damaged region using their information to reconstruct the occlusion. Finally, to those unsatisfactory results due to the lack of information we use the deep neural network interpolation algorithm to achieve a coherent effect. The proposed method has been tested on different types of vintage film sequences and proved to be effective.

This paper presents an image inpainting algorithm which relies on the fast global optimal searching. The whole framework of our method is focused on approximating the minimum of the proposed energy function with additional new texture term. We combine four steps in the framework. First, the initialization is carried out layer by layer in a fast way. Second, we use PatchMatch algorithm to realize the search of approximate nearest neighbors for patches in the missing region. Third, a weighted mean based image reconstruction method is implemented iteratively. Last, we use Poisson editing to handle the illumination of missing region to get realistic looking image. In addition, the iterative calculation is embedded in a multi-resolution pyramid. Experimental results show that our algorithm can effectively maintain the texture and structure characteristics of the missing region.

Taking the corrupted old film as the research object, this paper proposed a new video restoration method based on improved PatchMatch and low-rank matrix recovery. Our method is divided into three steps. Firstly, we divide each frame in the video sequence into image patches with overlap region, and similar interframe patches are found using the proposed improved PatchMatch algorithm. Then, the low-rank matrix recovery is used to separate the patch group into low-rank matrix component and sparse error component. Finally, synthesizing the video frame by the recorded location of patches, and completing the multi-frame joint automatic restoration frame by frame. The proposed method has been tested on a set of old film sequences in this paper. Experiment demonstrates that it is an effective method for corrupted old film restoration.