Visual Place Recognition (VPR) is a major challenge for robotics and autonomous systems, with the goal of pre dicting the location of an image based solely on its visual features. State-of-the-art (SOTA) models extract global descriptors using the powerful foundation model DINOv2 as a backbone. These models either explore the cross-image correlation or propose a time-consuming two-stage re-ranking strategy to achieve better performance. However, existing works only utilize the final output of DINOv2, and the current cross-image correlation causes unstable retrieval results. To produce both discriminative and stable global descriptors, this paper proposes a stable cross-image correlation enhanced model for VPR called SciceVPR. This model explores the full poten tial of DINOv2 in providing useful feature representations that implicitly encode valuable contextual knowledge. Specifically, SciceVPR first uses a multi-layer feature fusion module to capture increasingly detailed task-relevant channel and spatial information from the multi-layer output of DINOv2. The designed module not only en hances local feature quality but also reduces feature extraction time by approximately 50 % compared to existing adaptations. Secondly, SciceVPR considers the invariant correlation between images within a batch as valuable knowledge to be distilled into the proposed invariant feature projector. After knowledge distillation, SciceVPR is able to produce a self-enhanced global descriptor from a single input that achieves performance comparable to a multi-frame correlated global descriptor. These two innovations enable SciceVPR to produce fairly robust global features regardless of domain shifts (e.g., changes in illumination, weather and viewpoint between pictures taken in the same place). Experimental results demonstrate that the base variant, SciceVPR-B, achieves SOTA performance on several challenging urban benchmarks. The large variant, SciceVPR-L, performs on par with SOTA two-stage models, scoring over 3 % higher in Recall@1 compared to existing models on the challenging Tokyo24/7 dataset. Our code is available at https://github.com/shuimushan/SciceVPR.

Using images captured by Unmanned Aerial Vehicles (UAVs) to perform 3D reconstruction is a cost-effective way to acquire high-quality 3D models for large-scale urban scenes. The challenge, however, has become choosing camera viewpoints and planning flight path accordingly. Existing methods either plan the aerial path heuristically or train a reconstructability predictor, where the accuracy and completeness losses are optimized separately in a two-phase approach, leading to inaccurate reconstruction results and poor generalizability. To address these issues, this paper proposes a dual-task learning framework that establishes the correlation between viewpoint poses and reconstruction quality. In particular, the reconstructability estimation problem is modeled as two subtasks: reconstruction accuracy and reconstruction completeness, allowing both subtasks to be tackled simultaneously within a unified network. The model’s generalizability is improved by the soft parameter sharing and a new dual-loss function with trainable weight parameters. In addition, an adaptive viewpoint optimization strategy is proposed to refine an initial set of viewpoints generated based on the learned reconstructability. Our framework is extensively evaluated on both public datasets and two datasets we collected. Qualitative and quantitative experimental results demonstrate the superiority of our method in both synthetic and real scenes. Our framework achieves consistent improvements over state-of-the-art approaches by an average of 3.6% in F-score with 15% fewer images, and surpasses Oblique Photography with a 6% F-score gain while using 40% less image data. These advancements hold universally across all test scenes, outperforming prior methods in terms of accuracy and completeness.

Text-video retrieval establishes accurate similarity relationships between text and video through feature enhancement and granularity alignment. However, relying solely on similarity to associate intra-pair features and distinguish inter-pair features is insufficient, \textit{e.g.}, when querying a multi-scene video with sparse text or selecting the most relevant video from many similar candidates. In this paper, we propose a novel Dual Uncertainty Quantification (DUQ) model that separately handles uncertainties in intra-pair interaction and inter-pair exclusion. Specifically, to enhance intra-pair interaction, we propose an intra-pair similarity uncertainty module to provide similarity-based trustworthy predictions and explicitly model this uncertainty. To increase inter-pair exclusion, we propose an inter-pair distance uncertainty module to construct a distance-based diversity probability embeding, thereby widening the gap between similar features. The two components work synergistically, jointly improving the calculation of similarity between features. We evaluate our model on six benchmark datasets: MSRVTT (51.2%), DiDeMo, MSVD, LSMDC, Charades, and VATEX, achieving state-of-the-art retrieval performance.

Aquatic species play crucial roles in global ecosystems but are increasingly threatened by factors such as overfishing, coastal development and climate change. Existing deep learning methods address these challenges by employing powerful networks and large-scale, diverse datasets, separately tackling species recognition and trait identification during ongoing monitoring. However, they often exhibit limited generalization ability. Inspired by the human ability to quickly identify fish species and their locations with just a glance at an underwater image or scene, we introduce FishDetectLLM-a framework built on the lightweight TinyLLaVA architecture. FishDetectLLM utilizes the powerful reasoning capabilities and vast world knowledge of large language models (LLMs) to address the fish detection problem, providing both fish classification results and predicted bounding boxes for fish. Specifically, we create instruction dialogues for fish detection that connect fish taxonomy with classification descriptions and map location descriptions to the corresponding coordinates of bounding box in the input images from the recently released large-scale FishNet dataset. Then, we pretrain and fine-tune FishDetectLLM to achieve fish detection using the created dataset, leveraging the principle of augmenting human knowledge. Our results show that FishDetectLLM significantly outperforms existing multimodal LLMs and task-specific methods. Unlike conventional detection architectures that struggle to generalize beyond the training data, FishDetectLLM exhibits strong generalization capabilities, achieving robust performance on unseen data. This innovation paves the way for future applications of MLLMs in full research and offers valuable tools for the conservation of fish biodiversity.

Underwater images often suffer from color distortion and detail loss due to the scattering and absorption of light, presenting significant challenges in Underwater Image Enhancement (UIE). Although wavelet-based learning methods address this problem by correcting colors in low-frequency components and enhancing details in high-frequency components, they still struggle to achieve visual fidelity for human perception. Asa perceptually driven approach, conditional Denoising Diffusion Models (CDDMs) combined with wavelet transforms have been widely adopted for UIE. However, these methods often focus on the generative capability of CDDM in the low-frequency components, while neglecting the effectiveness of CDDM in high-frequency processing as well as the role of accurate priors in guiding the diffusion process. To address these limitations, we propose Dual-Branch Wavelet Diffusion models with Dual-Prior Refinement (DwaveDiff) for UIE. By decomposing the image into low-frequency and high-frequency subbands using the Haar wavelet transform, the reduced-dimensional frequency information not only accelerates CDDM inference but also provides distinct subbands, allowing CDDM to effectively handle color correction and detail recovery separately. Specifically, we use the Red Channel Prior image as a condition for the low-frequency branch of the CDDM to correct color, and the Edge Captured Map as a condition for the high-frequency branch of the CDDM to recover details. In addtion, the prior refinement strategy in the CDDM ensures that accurate prior information is used, guiding DwaveDiff to perform effective enhancement. Experimental results on both synthetic and real-world image datasets demonstrate that our method outperforms existing approaches both quantitatively and qualitatively.

Image dehazing is an important preliminary step for downstream vision tasks. Existing deep learning-based methods have limited generalization capabilities for real hazy images because they are trained on synthetic data and exhibit high domain-specific properties. This work proposes a new Diffusion Model for Synthetic-to-Real dehazing (DMSR) based on the haze-aware density. DMSR mainly comprises of a physics-based dehazing model and a Conditional Denoising Diffusion Model (CDDM)-based model. The coarse transmission map and coarse dehazing result estimated by the physics-based dehazing model serve as conditions for the subsequent CDDM-based model. In this process, the CDDM-based dehazing model progressively refines the coarse transmission map while generating the dehazing result, enabling the model to remove haze with accurate haze density information. Next, we propose a haze density-aware resampling strategy that incorporates the coarse dehazed result into the resampling process using the transmission map, thereby fully leveraging the diffusion model for heavy haze removal. Moreover, a new synthetic-to-real training strategy with the prior-based loss function and the memory loss function is applied to DMSR for improving generalization capabilities and narrowing the gap between the synthetic and real domains with low computational cost. Extensive experiments on various real datasets demonstrate the effectiveness and superiority of the proposed DMSR over state-of-the-art methods.

Underwater images suffer from different types of degradation, where color degradation occurs in the spatial domain and edge degradation in the frequency domain. The high-quality underwater image enhancement represents a crucial milestone in advancing computer vision systems tailored for marine environments. This foundational endeavor encompasses a wide array of applications in computer vision tasks, including underwater inspection, underwater archaeology, and environmental monitoring. However, current convolutional neural network (CNN)-based pyramid frameworks primarily focus on capturing local features, often overlooking the significance of global semantic features that play a crucial role in understanding underwater scenes. Moreover, these frameworks handle spatial and frequency features independently, failing to enhance images by exploring correlation among domain-specific attributes for enabling information consistency. Besides, optimizing the model using a loss function with the same domain attributes from the ground truth may not lead to a better generalization ability. To solve these problems, we propose a new Convolution-Transformer Blend Pyramid Network (CTPN), which consists of a spatial branch and several frequency branches. The CTPN has four key components: a Swin transformer encoder, a CNN-Transformer aggregated encoder–decoder (CTED) and a blend pyramid framework. The Swin transformer encoder is employed to capture global semantic features, benefiting from its ability to extract long-range and global dependencies among features. The CTED fuses local features captured by CNN layers and global semantic features captured by the Swin transformer encoder in the spatial branch, with the help of the Cross-Model Fusion Module (CFM) and Skip-Aggregation Module (SAM). Subsequently, a blend pyramid framework is designed which not only progressively expands the transformed information of the previous domain branch to the current domain branch via the CTED-based refining operation, but also utilizes the proposed Domain Affinity Block (DAB) to explore the connection between domain attributes, ensuring information consistency. The experimental results demonstrate that the proposed method outperforms existing underwater image enhancement methods quantitatively and qualitatively.

Occlusions and complex backgrounds are common factors that hinder many computer vision applications. In a street scene, the challenge of accurately predicting pedestrian trajectories comes from the complexity of human behavior and the diversity of the external environment. It is difficult, if not impossible, to extract relevant information to accurately predict pedestrian trajectories in dynamic scenes. Synthetic aperture imaging (SAI) uses an array of cameras to mimic a camera with a large virtual convex lens by projecting images of a scene from different views onto a virtual focal plane. It is commonly used to reconstruct occluded objects, and in a street scene, can provide observation of pedestrians occluded by other objects and pedestrians. In this paper, we propose a joint prediction method based on autofocusing of SAI to predict pedestrian trajectories in dynamic scenes. The main contributions of this paper include: 1) The task of pedestrian trajectory prediction in dynamic scenarios is redefined as pedestrian trajectory prediction and SAI autofocusing from a practical but more challenging perspective. 2) The proposed method is based on an existing SAI-based method to extract information in heavily occluded views, which can obtain more accurate results but with less computational cost and without using other sensors such as LiDAR or depth cameras. 3) A new pedestrian trajectory prediction model, an attention-based trajectory prediction variational autoencoder (ATP-VAE), is proposed to extract complex human behavior and social interactions in dynamic scenes through a new Intention Attention Unit. The experimental results on multiple public datasets show that the proposed method achieves state-of-the-art results in the first-person perspective and in aerial view.

Given a 3D mesh, we aim to synthesize 3D textures that correspond to arbitrary textual descriptions. Current methods for generating and assembling textures from sampled views often result in prominent seams or excessive smoothing. To tackle these issues, we present TexGen, a novel multi-view sampling and resampling framework for texture generation leveraging a pre-trained text-to-image diffusion model. For view consistent sampling, first of all we maintain a texture map in RGB space that is parameterized by the denoising step and updated after each sampling step of the diffusion model to progressively reduce the view discrepancy. An attention-guided multi-view sampling strategy is exploited to broadcast the appearance information across views. To preserve texture details, we develop a noise resampling technique that aids in the estimation of noise, generating inputs for subsequent denoising steps, as directed by the text prompt and current texture map. Through an extensive amount of qualitative and quantitative evaluations, we demonstrate that our proposed method produces significantly better texture quality for diverse 3D objects with a high degree of view consistency and rich appearance details, outperforming current state-of-the-art methods. Furthermore, our proposed texture generation technique can also be applied to texture editing while preserving the original identity. More experimental results are available at https://dong-huo.github.io/TexGen/

Omnidirectional images have attracted significant attention in recent years due to the rapid development of virtual reality technologies. Equirectangular projection (ERP), a naive form to store and transfer omnidirectional images, however, is challenging for existing two-dimensional (2D) image super-resolution (SR) methods due to its inhomogeneous distributed sampling density and distortion across latitude. In this paper, we make one of the first attempts to design a spherical pseudo-cylindrical representation, which not only allows pixels at different latitudes to adaptively adopt the best distinct sampling density but also is model-agnostic to most off-the-shelf SR methods, enhancing their performances. Specifically, we start by upsampling each latitude of the input ERP image and design a computationally tractable optimization algorithm to adaptively obtain a (sub)-optimal sampling density for each latitude of the ERP image. Addressing the distortion of ERP, we introduce a new viewport-based training loss based on the original 3D sphere format of the omnidirectional image, which inherently lacks distortion. Finally, we present a simple yet effective recursive progressive omnidirectional SR network to showcase the feasibility of our idea. The experimental results on public datasets demonstrate the effectiveness of the proposed method as well as the consistently superior performance of our method over most state-of-the-art methods both quantitatively and qualitatively.

Due to the scattering of light and the influence of different water types, underwater images usually suffer from different type of hybrid degradation, e.g. color distortion, blurred details and low contrast. Existing underwater image enhancement methods are weak at handling hybrid degradation simultaneously, resulting in low quality results. Inspired by the fact that wavelet-based enhancement methods can correct color and enhance details in frequency domain and the color compensation prior can compensate missing color information in spatial domain, we design the Multi-level Wavelet-based Underwater Image Enhancement Network (MWEN) with the color compensation prior to enhance image in both frequency domain and spatial domain. Specifically, we integrate the multi-level wavelet transform and the color compensation prior into a multi-stage enhancement framework, where each stage consists of a Multi-level Wavelet-based Enhancement Module (MWEM), a Color Compensation Prior Extraction Module (CCPEM) and a color filter with prior-aware weights. The MWEM decomposes image features into low frequency and high frequency by a wavelet transform, and then enhances them by a low frequency enhancement branch and several high frequency enhancement branches, respectively. The low frequency reduces the color distortion of different water types using Instance Normalization for style transfer, while the high frequency enhancement enhances sparse details using a non-local sparse attention mechanism. After the inverse wavelet transform, the preliminary enhanced result by the MWEM is obtained. Then, the color filter whose weights are customized by the color compensation information extracted from the CCPEM dynamically is applied to output of the MWEM for color compensation. Such an operation enables network to adapt to hybrid degradation and achieve better performance. The experiments demonstrate MWEN outperforms existing UIE methods quantitatively and qualitatively.

This paper tackles spectral reflectance recovery (SRR) from RGB images. Since capturing ground-truth spectral reflectance and camera spectral sensitivity are challenging and costly, most existing approaches are trained on synthetic images and utilize the same parameters for all unseen testing images, which are suboptimal especially when the trained models are tested on real images because they never exploit the internal information of the testing images. To address this issue, we adopt a self-supervised meta-auxiliary learning (MAXL) strategy that fine-tunes the well-trained network parameters with each testing image to combine external with internal information. To the best of our knowledge, this is the first work that successfully adapts the MAXL strategy to this problem. Instead of relying on naive end-to-end training, we also propose a novel architecture that integrates the physical relationship between the spectral reflectance and the corresponding RGB images into the network based on our mathematical analysis. Besides, since the spectral reflectance of a scene is independent to its illumination while the corresponding RGB images are not, we recover the spectral reflectance of a scene from its RGB images captured under multiple illuminations to further reduce the unknown. Qualitative and quantitative evaluations demonstrate the effectiveness of our proposed network and of the MAXL. Our code and data are available at https://github.com/Dong-Huo/SRR-MAXL.

Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Copy DOI

While recent advancements in deep-learning point cloud upsampling methods have improved the input to intelligent transportation systems, they still suffer from issues of domain dependency between synthetic and real-scanned point clouds. This paper addresses the above issues by proposing a new ray-based upsampling approach with an arbitrary rate, where a depth prediction is made for each query ray and its corresponding patch. Our novel method simulates the sphere-tracing ray marching algorithm on the neural implicit surface defined with an unsigned distance function (UDF) to achieve more precise and stable ray-depth predictions by training a point-transformer-based network. The rule-based mid-point query sampling method generates more evenly distributed points without requiring an end-to-end model trained using a nearest-neighbor-based reconstruction loss function, which may be biased towards the training dataset. Self-supervised learning becomes possible with accurate ground truths within the input point cloud. The results demonstrate the method's versatility across domains and training scenarios with limited computational resources and training data. Comprehensive analyses of synthetic and real-scanned applications provide empirical evidence for the significance of the upsampling task across the computer vision and graphics domains to real-world applications of ITS.

The use of MR images in medical image analysis from different centers in clinical applications and medical research has grown in popularity. However, challenges arise due to inherent variability between centers, leading to domain shift issues that reduce the reliability and robustness of the analysis results. Furthermore, the lack of suitable tools for analyzing domain shift hampers the progress of developing and validating domain adaptation and harmonization techniques. Utilizing pre-trained deep models as feature extractors, we introduce a novel framework called Deep Domain Shift analyzer for MRI (DeepDSMRI), designed explicitly to comprehend the extent of domain shift in MRI datasets. DeepDSMRI provides adequate insights into the existence of domain shift for diverse MRI modalities, including structural, functional, and diffusion-weighted images. The proposed framework incorporates visualization tools (e.g., t-SNE and UMAP) to illustrate grouping similar data and isolating dissimilar data into distinct clusters. Moreover, the quantitative analysis measures the classification accuracy between domains and the domain shift distance. The efficacy of the proposed DeepDSMRI is demonstrated through experimental assessments conducted on seven extensive multi-center neuroimaging databases. The source code is available at (https://github.com/rkushol/DeepDSMRI).

The key to crossview geolocalization is to match images of the same target from different viewpoints, e.g., images from drones and satellites. It is a challenging problem due to the changing appearance of objects from variable viewpoints. Most existing methods focus mainly on extracting global features or on segmenting feature maps, causing the loss of information contained in the images. To address the above issues, we propose a new ConvNeXt-based method called MCCG, which stands for Multiple Classifier for Cross-view Geolocalization. The proposed method captures rich discriminative information by cross-dimension interaction and acquires multiple feature representations, realizing a comprehensive feature representation. Additionally, the robustness of the model is improved crediting the multiple feature representations exploiting more contextual information despite position shifting or scale variations. Extensive experiments on the widely used public benchmarks University-1652 and SUES-200 demonstrate that the proposed method achieves state-of-the-art performance in both drone-view target localization and drone navigation applications by over 3% compared to existing methods. Our code and model are available at https://github.com/mode-str/crossview .

Image dehazing is an improtant image pre-processing step for many computer vision tasks with many proposed methods using convolutional neural networks. Ordinary Differential Equations (ODE) are a powerful mathematical tool in social and natural science, leading to better understanding, prediction, and use of information systems. Because prior knowledge has demonstrated its effectiveness in many practical domains, we design a Visual Attention Network (VAN) and an ODE-inspired Network (ODEN) based on prior knowledge. Then, we develop a multi-model fuzzy fusion strategy, which integrates results predicted by the prior-based Visual Attention Network (VAN) and the ODE-inspired Network (ODEN) into a single network to leverage their respective strengths in improving the dehazing performance. First, to utilize the haze-related prior for dehazing, the VAN removes haze with the help of the haze attention map. Then, to improve performance of stacked residual blocks inspired by the first-order Euler method in ODE-based methods, the ODEN is built by only 3 Runge–Kutta Modules (RKM), each of which not only is related to the stable fourth-order Runge–Kutta method implemented in the Runge–Kutta Block (RKB), but also combines the RKB with an attention mechanism. Finally, an attention based fusion mechanism is used to fuse results estimated by the ODEN and the VAN based on the multi-level features extracted by a pretrained ResNeXt. The experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods in terms of visual effects and accuracy. The average PSNR and SSIM on three public datasets are 31.36 and 0.9766, respectively, which are better than the compared state-of-the-art methods.

Transformer-based architectures start to emerge in single image super resolution (SISR) and have achieved promising performance. However, most existing vision Transformer-based SISR methods still have two shortcomings: (1) they divide images into the same number of patches with a fixed size, which may not be optimal for restoring patches with different levels of texture richness; and (2) their position encodings treat all input tokens equally and hence, neglect the dependencies among them. This paper presents a HIPA, which stands for a novel Transformer architecture that progressively recovers the high resolution image using a hierarchical patch partition. Specifically, we build a cascaded model that processes an input image in multiple stages, where we start with tokens with small patch sizes and gradually merge them to form the full resolution. Such a hierarchical patch mechanism not only explicitly enables feature aggregation at multiple resolutions but also adaptively learns patch-aware features for different image regions, e.g., using a smaller patch for areas with fine details and a larger patch for textureless regions. Meanwhile, a new attention-based position encoding scheme for Transformer is proposed to let the network focus on which tokens should be paid more attention by assigning different weights to different tokens, which is the first time to our best knowledge. Furthermore, we also propose a multi-receptive field attention module to enlarge the convolution receptive field from different branches. The experimental results on several public datasets demonstrate the superior performance of the proposed HIPA over previous methods quantitatively and qualitatively. We will share our code and models when the paper is accepted.