Dense video captioning is a multimodal task that involves visual feature extraction and natural language generation. Previous approaches face several challenges, such as reliance on complex "localize-then-describe" frameworks, limited contextual information, and the potential for inaccurate detection. In this paper, we propose an end-to-end dense video captioning model that leverages multimodal data and multi-level visual features, based on the pretrained ViFi-CLIP model. Our approach constructs an integrated network that unifies video segmentation and caption generation into a single end-to-end framework, effectively utilizing visual, audio, and textual modalities. In selecting visual features, we comprehensively consider the impact of grid and region-based features and employ ViFi-CLIP to map these features into a joint vision–language embedding space, thereby enhancing the semantic relevance of the visual representations. Extensive experiments conducted on the ActivityNet dataset demonstrate the effectiveness of our model. Results show that our approach outperforms PDVC by 8–10
We present a comparative study of segmentation methods for high-power laser applications, focusing on two specific challenges: detection of microscopic surface damage on optical components and detection of radiochromic films for reconstructing high-dimensional particle phase space distributions. Both applications involve complex morphological variations and non-homogeneous contrast conditions, requiring robust and scalable analysis methods. We evaluate two conventional algorithms and two deep learning-based instance segmentation models, including YOLOv8n-seg and a Detectron2-based Mask R-CNN implementation. All models are evaluated on real datasets that reflect the experimental complexities. We focus particular attention to the accuracy of contour detection, using geometric evaluation metrics such as radial contour comparison, Hausdorff distance, Chamfer distance, as well as intersection-over-union, and analysing runtime performance. Our results indicate that the YOLOv8n-seg model outperforms the conventional surface damage segmentation method in accuracy, but with 12 times higher computational requirements. In contrast, for radiochromic films analysing YOLOv8n-seg achieves both higher accuracy and faster evaluation. In comparison to YOLOv8n-seg model, Detectron2-based Mask R-CNN implementation lags in both segmentation performance and runtime. These results highlight the potential of YOLOv8n-seg model in addressing specific data-related challenges in modern laser diagnostics and support their role in the development of next-generation automated analysis systems.
Optical remote sensing images have unique advantages in the marine environment, but traditional target detection algorithms have limited performance due to complex background, small target size and dense target distribution. In this study, we propose the YOLOv5–BSD algorithm based on YOLOv5, which optimises the feature fusion network and combines a weighted feature pyramid network and a novel detection head to improve the detection accuracy. The SIoU loss function with angular penalty is introduced to reduce the degree of freedom, and deformable convolution is used to improve the generalization ability. Experiments on the ShipRSImageNet V1.0 data set show that YOLOv5–BSD improves the mAP@0.5, mAP@0.5:0.95, precision and recall by 2.5, 3.7, 2.48 and 2.15 percentage points, respectively, which verifies the effectiveness of the optimization strategy.
Virtual try-on technology has emerged as a key driver of online clothing sales growth. It addresses the challenge of not being able to try on garments in person, thus improving the online shopping experience. This paper provides a comprehensive review of deep learning-based virtual try-on research. It organizes relevant literature across several areas. These include feature extraction techniques, clothing warping technologies, image synthesis methods, benchmark data sets, and evaluation metrics. This paper also offers a summary and comparative analysis of existing approaches. Special emphasis is given to tasks, such as spatial-semantic alignment between garments and human poses, preservation of clothing texture features, and high-fidelity image generation. Finally, this paper elaborates on current limitations of virtual try-on technology and proposes directions for future research. The goal is to support the broader application of this technology in e-commerce.
Biometrics has become a viable and popular solution for applications which require secure authentication. In spite of the advantages of biometrics as an automatic authentication technology, the usage of biometric characteristics raises significant concerns regarding personal data and privacy of subjects in these systems. To address these concerns, several biometric template protection schemes have been proposed in the literature to achieve trustworthy, reliable and privacy-preserving systems. In this paper, we benchmark several cancelable biometrics (CB) schemes on different biometric characteristics. We consider BioHashing, Multi-Layer Perceptron (MLP) hashing, Bloom filters, and two schemes based on Index-of-Maximum (IoM) hashing (i.e. IoM-URP and IoM-GRP). In addition to the mentioned CB schemes, we introduce a CB scheme (as a baseline) based on user-specific random transformations followed by binarization. We evaluate the unlinkability, irreversibility, and recognition performance (which are the required criteria by the ISO/IEC 24745 standard) of these CB schemes on deep learning-based templates extracted from different physiological and behavioural biometric characteristics including face, voice, finger vein, and iris. Our experiments show that all the studied CB schemes are almost unlinkable for different characteristics. We also observe that the mutual information (MI) between protected and unprotected templates varies according to the scenario and biometric characteristic. In terms of recognition accuracy, our study shows that deep templates protected by Bloom filters suffer from a drop in performance, while other CB schemes achieve competitive accuracies for different biometric characteristics. We provide an open-source implementation of all the experiments presented to facilitate the reproducibility of our results: https://github.com/otroshi/benchmark_cb .
Performing video restoration in embedded systems is a challenge. Indeed, state-of-the-art learning-based methods in video restoration are specifically trained for given tasks, and often rely on huge networks, which makes them unsuitable when memory and computing resources are limited. In this paper, we explore the use of deep Plug-and-Play (PnP) algorithms for video restoration. We distinguish ourselves from prior PnP work by directly writing PnP schemes on video sequences instead of separate images. Our experiments in video deblurring, super-resolution (SR), and pixel interpolation all show a clear benefit to using a network specifically designed for video denoising, as it yields better restoration performance than a single image network with similar denoising performance using the same PnP formulation, and more temporal stability than other image-based PnP works. Experiments also highlight that, while not always as effective as state-of-the-art single-task deep networks, our lightweight approach remains competitive for many video restoration tasks.
Image quality assessment (IQA) is a field that focuses on evaluating the quality of images, playing a crucial role in various image processing and/or computer vision applications. Traditional full-reference (FR) IQA algorithms struggle with an accurate perceptual quality evaluation due in part to their reliance on handcrafted features and simple mathematical functions to calculate the elementwise distance between the reference and distorted images. Although deep-learning-based FR IQA methods have shown advantages in providing a certain degree of tolerance to texture resampling, their performances are still limited by the redundant model parameters and ineffective quality-aware feature extraction/representation. To address this issue, in this paper, we propose a multi-modal dual-attention FR IQA algorithm based on combining a global-and-local image structure analysis with text information interpreted by the widely used large language model. Specifically, the proposed multi-modal dual-attention network consists of four modules. First, a global-and-local feature extraction module was employed to extract the quality-aware features from the reference and distorted images, which were then realigned along the spatial and channel dimensions by a feature fusion module. To take into account both the channel and spatial attentions and thus increase the model capacity in representing long-range dependencies among different image areas, a feature enhancement module was designed to encode the spatial information along two directions, based on which the direction-aware attention maps with position information were generated. Finally, the text prior knowledge interpreted by the contrastive language-image pre-training (CLIP) model was embedded to assist the attention-based prediction module for quality estimation. Experimental results on four benchmark datasets demonstrate the effectiveness of our model as compared with other state-of-the-art FR IQA methods. The code is available at https://vinelab.jp/m2da/.
Acoustic event detection faces great challenges in recognizing event categories and locating event temporal boundaries due to the scarcity of high-quality labeled data and the limitation of label granularity. To conquer these constraints, a novel data augmentation method, LTSRR, and a two-stage method, PCL-AED, are proposed in this study. LTSRR is developed to generate training samples with rich spectral characteristics, which significantly expands the diversity of the training data. Subsequently, PCL-AED puts up a segment-level progressive event classification strategy for alleviating the impact of uneven sample distribution on audio tagging, and a temporal contrast learning mechanism is designed to enhance the sensitivity of the model to event boundaries for acoustic event detection. Experiments on DCASE2018 and DCASE2021 Task 4 datasets show that the LTSRR and PCL-AED proposed in this paper exhibit competitive performance, providing a new stroke to the semi-supervised acoustic event detection task.
Single image super-resolution (SISR) plays a crucial role in enhancing image quality for resource-constrained devices, particularly in biomedical imaging and point-of-care diagnostics. However, existing SISR methods face a fundamental trade-off: high-performance models achieve excellent reconstruction quality but require substantial computational resources, while lightweight approaches significantly reduce complexity at the cost of notable performance degradation. To address this challenge, this paper proposes HOLI-SRNet, a unified lightweight SISR framework specifically designed for resource-constrained deployment. The framework integrates three synergistic components: (1) COMPASS—a comprehensive pruning and sparsification strategy that reduces parameter redundancy through hybrid structured/unstructured pruning while preserving essential features; (2) dual-attention mechanisms—attention-based adaptive residual block (AARB) for dynamic kernel selection and Attention-based cross-layer connection (ACC) for adaptive multi-scale feature fusion; (3) hardware–software co-optimization—specialized operator libraries and intelligent scheduling mechanisms that bridge the gap between algorithmic efficiency and hardware-specific acceleration. Experimental results demonstrate HOLI-SRNet’s effectiveness in balancing performance and efficiency: the framework achieves competitive reconstruction quality (32.29dB PSNR) while requiring 96
Forward adaptive transform coding of images requires a codebook of transform matrices from which the best transform can be chosen for each macroblock. Codebook construction is a problem of designing a quantizer for Karhunen–Lóeve transform (KLT) matrices estimated from sample image blocks. We present a novel method for KLT matrix quantization based on a finite-lattice non-causal homogeneous Gauss–Markov random field (GMRF) model with asymmetric Neumann boundary conditions for blocks in natural images. The matrix quantization problem is solved in the GMRF parameter space, simplifying the harder problem of quantizing a large matrix subject to an orthonormality constraint to a low-dimensional vector quantization problem. Typically used GMRF parameter estimation methods such as maximum-likelihood (ML) do not necessarily maximize the coding performance of the resulting transform matrices. To this end we propose a method for GMRF parameter estimation from sample image data, which maximizes the high-rate transform coding gain. We also investigate the application of GMRF-based transforms to variable block-size adaptive transform coding.
Novel view synthesis of dynamic 3D scenes is an attractive but challenging problem. Many recent methods extend the 3D Gaussian Splatting model with temporal attributes in order to achieve high-quality and real-time rendered dynamic 3D scene representation. In order to have a more complete understanding of 4D scenes, we propose Spacetime Gaussian Grouping (SGG), for segmenting and tracking the objects spatially and temporally. The proposed method trains the Spacetime Gaussians (STG) model in conjunction with the multi-view consistent segmentation masks corresponding to the input images, which are used to label the 3D Gaussians with instance identities. Our model produces a lightweight, dynamic 3D representation of a scene, enabling users to select and edit specific objects in a 4D manner across various devices, including smartphones, PCs, VR systems, among other devices. Furthermore, a novel dynamic view synthesis segmentation benchmark is proposed to evaluate our method quantitatively.
Detecting abnormal events in videos is essential for effective anomaly detection in surveillance environments. In this work, we propose a framework that combines the Abnormality-Aware Fused Attention Model (AAFAM) with the Global Density Joined Network (GDJNet) to enhance abnormal event detection. AAFAM uses spatial and channel-wise attention mechanisms to focus on anomalous regions in feature maps. It suppresses irrelevant background information. GDJNet, on the other hand, captures both local and global spatial relationships through density estimation. It allows the model to learn object distributions and co-occurrence patterns at multiple scales. To strengthen the model’s performance, we integrate AAFAM and GDJNet using a fusion strategy that combines attention and density maps, resulting in highly discriminative feature representations. Experimental results show that the proposed AAFAM–GDJNet framework outperforms existing methods and achieves state-of-the-art performance.
Since Spatial Scalable High Efficiency Video Coding (SSHVC) utilizes multi-layer encoding and inter-layer prediction, it has extremely high coding complexity which has severely hindered its wide spread. In order to improve coding speed, we develop a fast intra coding algorithm to improve coding speed of SSHVC. First, we utilize the textural feature to obtain the probabilities of using Coding Units (CUs), so as to predict candidate CUs and exclude unlikely ones. Second, we jointly utilize the textural features, the neighboring Coding Modes (CMs) and Rate Distortion (RD) cost to obtain the probability of Intra Mode (IM), which is then combined with the probability of its collocated CU to exclude unlikely IMs. Finally, we investigate distribution of Direction Modes (DMs), which are then combined with the probability of their collocated CU to predict candidate DMs and exclude unlikely ones. Experimental results demonstrate that the proposed algorithm can significantly improve coding speed by 78.64
This paper introduces a novel approach to balance the accuracy and efficiency in Convolutional Neural Networks (CNNs). The main contribution is a dual strategy that exploits feature reuse along a squeezed architecture. The idea is to integrate fire units from the SqueezeNet architecture, replacing standard convolutions to reduce the number of trainable parameters, and introducing data reuse through additional bypass connections to maintain accuracy. While the idea can be broadly applicable, we demonstrate its effectiveness by applying it to the UNet architecture, leading to the development of a novel network called Squeeze-Bypass-UNet (SBp-UNet). The network is therefore used for the challenging task of pedestrian semantic segmentation, particularly in scenarios involving heavy occlusions. We validate our model on the Occluded Human (OCHuman) dataset, demonstrating superior performance to the original UNet with 9.33 times fewer parameters. Further validation on the Pascal VOC dataset confirms the robust predictive ability of SBp-UNet. Our work thus offers a novel solution for the efficiency–accuracy problem in CNNs, with proven application in pedestrian segmentation.
The work is focused on video compression for the scenarios, where the decoded video serves not only human viewers but also as input for systems implementing various machine vision tasks, such as object detection and tracking. The proposed innovative tool is based on retargeting video frames processed based on Regions of Interest (RoI), corresponding to the individual objects detected in the frames. Experimental evaluation demonstrates significant average bitrate reduction while maintaining the same quality, ranging from 3 to 57% depending on the machine vision task and encoding scenario. The proposal underwent thorough consideration within the MPEG group and was adopted for the upcoming Video Coding for Machines (VCM) technology.
Video super-resolution, which involves improving the spatial resolution of low-resolution video sequences, plays a pivotal role in computer vision. The use of regularization methods, incorporating various mathematical constraints, is crucial for enhancing the quality and visual clarity of super-resolved videos. In this study, we introduce a new technique for video super-resolution that incorporates an innovative denoiser within the ADMM algorithm. Our findings demonstrate the superiority of our approach over several state-of-the-art methods.
The modern industrial equipment are widely used in various industries. Its performance is notably affected by temperature variations, making temperature monitoring crucial. To achieve this, we employ a combination of optical and infrared sensors to visualize and measure the temperature field. This paper presents a new infrared and visible image segmentation forward model, along with a Bayesian estimation framework featuring a specially designed Markov-Potts prior model. This model addresses uncertainties in the forward model errors of both infrared and visible images. We use a joint maximum a posteriori (JMAP) criterion and an alternate optimization algorithm to estimate both fused image, its segmentation, and all the parameters of the probabilistic model. The proposed method is tested on benchmark datasets and industrial scenarios, demonstrating its robustness and effectiveness.
Due to the rapid advances in computer vision and deep learning, human action recognition has become one of the most important representative tasks for video understanding. Especially for human action recognition based on RGB-D data, a promising research direction, there has been a number of researchers to work on. In particular, convolutional neural networks (CNNs) are capable of image classification tasks, recurrent neural networks (RNNs) are skilled in sequence-based problems, and Transformer is good at global modeling. In this survey, we introduce a number of algorithms based on CNNs, RNNs and Transformer for RGB-D based human action recognition, which could be categorized into four parts: RGB-based, depth-based, skeleton-based and RGB-D based. As a survey focusing on the RGB-D based human action recognition, we thoroughly represent the algorithms, datasets and popular applications for it. What’s more, we give some possible future research directions for this field in the last part.
Infrared and visible image fusion aims to generate fused images that maintain the advantages of each source such as temperature information and detailed textures. This paper presents Bayesian Model-based Fusion-Net, a novel approach for infrared and visible image fusion. By formulating image fusion as an inverse problem within a hierarchical Bayesian framework, our method leverages physical priors and data-driven techniques to enhance model interpretability and transferability. Compared to traditional and deep learning-based fusion methods, the proposed Bayesian Model-based Fusion-Net achieves promising performance with significantly reduced computational complexity (0.07G FLOPs). Extensive experiments on multiple datasets, including industrial public dataset, demonstrate the effectiveness of the proposed method in preserving texture details, maintaining structural integrity, and enhancing feature clarity. Furthermore, our approach exhibits robustness when trained with limited data, maintaining consistent performance even when using only 10% of the training dataset. These characteristics make the proposed Bayesian Fusion-Net particularly suitable for industrial monitoring applications where computational resources and the amount of training dataset are limited.
In cryptocurrency transactions, effective wallet key management by users is crucial for asset protection. Fuzzy Signature (FS) schemes enhance the binding between signers and their keys by using biometric information instead of traditional signing keys, offering secure and convenient key management. However, FS faces two significant privacy concerns. First, using biometric information as the signing key restricts users’ ability to manage multiple distinct keys for different applications. Second, the design that absorbs biometric fuzziness during verification results in publicly available verification keys that contain data dependent on biometric information. This paper introduces Biometrically-Independent Fuzzy Signature (BIFS), a variant addressing existing FS scheme problems. We propose a generic construction for BIFS and present a BIFS scheme compatible with BLS signatures supported by Ethereum 2.0. Experimental results confirm that the success rate of signature generation maintains the same authentication accuracy as the employed facial recognition system. Moreover, our results show that the proposed scheme is practical since the signature generation process requires approximately 2 ms.
