Contrastive Language Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative cluster assignment and classification. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image. In this paper, we propose a novel Multi-Label Cluster Discrimination method named MLCD to enhance representation learning. In the clustering step, we first cluster the large-scale LAION-400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual objects or attributes, we select the multiple closest centers as auxiliary class labels. In the discrimination step, we design a novel multi-label classification loss, which elegantly separates losses from positive classes and negative classes, and alleviates ambiguity on decision boundary. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval.
Learning feature correspondence is a foundational task in computer vision, holding immense importance for downstream applications such as visual odometry and 3D reconstruction. Despite recent progress in data-driven models, feature correspondence learning is still limited by the lack of accurate per-pixel correspondence labels. To overcome this difficulty, we introduce a new self-supervised scheme, imperative learning (IL), for training feature correspondence. It enables correspondence learning on arbitrary uninterrupted videos without any camera pose or depth labels, heralding a new era for self-supervised correspondence learning. Specifically, we formulated the problem of correspondence learning as a bilevel optimization, which takes the reprojection error from bundle adjustment as a supervisory signal for the model. To avoid large memory and computation overhead, we leverage the stationary point to effectively back-propagate the implicit gradients through bundle adjustment. Through extensive experiments, we demonstrate superior performance on tasks including feature matching and pose estimation, in which we obtained an average of 30% accuracy gain over the state-of-the-art matching models.
Vision transformers (ViTs) have emerged as a promising alternative to convolutional neural networks (CNNs) for various image analysis tasks, offering comparable or superior performance. However, one significant drawback of ViTs is their resource-intensive nature, leading to increased memory footprint, computation complexity, and power consumption. To democratize this high-performance technology and make it more environmentally friendly, it is essential to compress ViT models, reducing their resource requirements while maintaining high performance. In this paper, we introduce a new block-structured pruning to address the resource-intensive issue for ViTs, offering a balanced trade-off between accuracy and hardware acceleration. Unlike unstructured pruning or channel-wise structured pruning, block pruning leverages the block-wise structure of linear layers, resulting in more efficient matrix multiplications. To optimize this pruning scheme, our paper proposes a novel hardware-aware learning objective that simultaneously maximizes speedup and minimizes power consumption during inference, tailored to the block sparsity structure. This objective eliminates the need for empirical look-up tables and focuses solely on reducing parametrized layer connections. Moreover, our paper provides a lightweight algorithm to achieve post-training pruning for ViTs, utilizing second-order Taylor approximation and empirical optimization to solve the proposed hardware-aware objective. Extensive experiments on ImageNet are conducted across various ViT architectures, including DeiT-B and DeiT-S, demonstrating competitive performance with other pruning methods and achieving a remarkable balance between accuracy preservation and power savings.
We present a novel discrete optimization-based approach to generate downsampled versions of binary images that are guaranteed to have the same topology as the original, measured by the zeroth and first Betti numbers of the black regions, while having good similarity to the original image as measured by IoU and Dice scores. To our best knowledge, all existing binary image downsampling methods don't have such topology-preserving guarantees. We also implemented a baseline morphological operation (dilation)-based approach that always generates topologically correct results. However, we found the similarity scores to be much worse. We demonstrate several applications of our approach. First, generating smaller versions of medical image segmentation masks for easier human inspection. Second, improving the efficiency of binary image operations, including persistent homology computation and shortest path computation, by substituting the original images with smaller ones. In particular, the latter is a novel application that is made feasible only by the full topology-preservation guarantee of our method.
Scene text segmentation aims at cropping texts from scene images, which is usually used to help generative models edit or remove texts. The existing text segmentation methods tend to involve various text-related supervisions for better performance. However, most of them ignore the importance of text edges, which are significant for downstream applications. In this paper, we propose Edge-Aware Transformers, termed EAFormer, to segment texts more accurately, especially at the edge of texts. Specifically, we first design a text edge extractor to detect edges and filter out edges of non-text areas. Then, we propose an edge-guided encoder to make the model focus more on text edges. Finally, an MLP-based decoder is employed to predict text masks. We have conducted extensive experiments on commonly-used benchmarks to verify the effectiveness of EAFormer. The experimental results demonstrate that the proposed method can perform better than previous methods, especially on the segmentation of text edges. Considering that the annotations of several benchmarks (e.g., COCO_TS and MLT_S) are not accurate enough to fairly evaluate our methods, we have relabeled these datasets. Through experiments, we observe that our method can achieve a higher performance improvement when more accurate annotations are used for training. The code and datasets are available at https://hyangyu.github.io/EAFormer/.
In this paper, we tackle a new and challenging problem of text-driven generation of 3D garments with high-quality textures. We propose "WordRobe", a novel framework for the generation of unposed textured 3D garment meshes from user-friendly text prompts. We achieve this by first learning a latent representation of 3D garments using a novel coarse-to-fine training strategy and a loss for latent disentanglement, promoting better latent interpolation. Subsequently, we align the garment latent space to the CLIP embedding space in a weakly supervised manner, enabling text-driven 3D garment generation and editing. For appearance modeling, we leverage the zero-shot generation capability of ControlNet to synthesize view-consistent texture maps in a single feed-forward inference step, thereby drastically decreasing the generation time as compared to existing methods. We demonstrate superior performance over current SOTAs for learning 3D garment latent space, garment interpolation, and text-driven texture synthesis, supported by quantitative evaluation and qualitative user study. The unposed 3D garment meshes generated using WordRobe can be directly fed to standard cloth simulation animation pipelines without any post-processing.
Previous studies in deepfake detection have shown promising results when testing face forgeries from the same dataset as the training. However, the problem remains challenging when one tries to generalize the detector to forgeries from unseen datasets and created by unseen methods. In this work, we present a novel general deepfake detection method, called Curricular Dynamic Forgery Augmentation (CDFA), which jointly trains a deepfake detector with a forgery augmentation policy network. Unlike the previous works, we propose to progressively apply forgery augmentations following a monotonic curriculum during the training. We further propose a dynamic forgery searching strategy to select one suitable forgery augmentation operation for each image varying between training stages, producing a forgery augmentation policy optimized for better generalization. In addition, we propose a novel forgery augmentation named self-shifted blending image to simply imitate the temporal inconsistency of deepfake generation. Comprehensive experiments show that CDFA can significantly improve both cross-datasets and cross-manipulations performances of various naive deepfake detectors in a plug-and-play way, and make them attain superior performances over the existing methods in several benchmark datasets.
The viewing graph is a useful way to represent uncalibrated cameras and their geometric relationships: nodes correspond to cameras and edges represent fundamental matrices. By analyzing this graph, it is possible to establish if the problem is "solvable" in the sense that there exists a unique (up to a single projective transformation) set of cameras that are compliant with the given fundamental matrices. In this paper, we take several steps forward in the study of viewing graph solvability: we propose a new formulation of the problem that is more direct than previous literature, based on a formula that explicitly links pairs of cameras via their fundamental matrix; we introduce the new concept of "infinitesimal solvability", demonstrating its usefulness in understanding real structure from motion graphs; we propose an algorithm for testing infinitesimal solvability and extracting components of unsolvable cases, that is more efficient than previous work; we set up an open research question on the connection between infinitesimal solvability and solvability.
Action Quality Assessment (AQA) has emerged as a burgeoning topic due to its wide applicability across various domains. However, most existing approaches score action videos by focusing on the deep features of the entire video, which lacks a fine-grained understanding of human motions, thereby limiting the accuracy and interpretability of AQA. In this paper, we consider both video features and human poses to assess action quality accurately. Human poses can assist in locating the concerned spatial position in image frames, thereby providing a precise execution of specific actions. To achieve this goal, we propose an effective dual-stream spatial position-aware AQA framework that integrates both video and human pose features to evaluate the actions. Specifically, we design a spatial position-aware (SPA) module to enhance the representation of local features at the spatial position of the athlete. Extensive experiments demonstrate that our approach achieves good performances on common AQA datasets, validating the effectiveness of the proposed method.
Coral reefs are crucial for biodiversity and provide vital resources for humankind. But despite such a central role, they are confronted to increasing threats linked to climate change, pollution, and local stressors. To ensure effective conservation, efficient and scalable monitoring is key: this necessitates automated identification of benthic classes and their states on a large scale through semantic segmentation. However, segmentation of underwater videos is challenging, because of visual similarities between benthic classes, underwater distortions and limited available datasets, making it harder to create accurate and robust models. In this paper, we present a method for training a semantic segmentation model on a small dataset of video frames of coral scenes, by fine-tuning a large transformer model. Our approach uses transfer learning on the Segment Anything Model (SAM), incorporating specific training and prediction strategies. We benchmark our model against a CNN for semantic segmentation as a baseline. Our results demonstrate a substantial improvement in model performance, particularly for benthic classes that often appear as small objects and rarer classes, highlighting the potential of our approach in advancing coral reef mapping and monitoring.
While studying objects presented in paintings, art history specialists identify their significance, symbolic meaning and historical context. Analyzing big artistic collections can be very time-consuming for the specialists. The search could be relieved by using modern object detectors. However, object detectors have poor performance on artistic images. This problem could be solved by fine-tuning them on specialized annotated datasets. In this paper, we explore the possibilities of using open-vocabulary foundation models for dataset annotation in a semi-automated manner. We propose an approach for artistic dataset annotation for object detection task based on a small set of images annotated on image-level and using Vision Transformer for Open-World Localization (OWL-ViT2) model, the YOLO object detector and an approximate nearest neighbour oh yeah (ANNOY) algorithm. We extend the existing DEArt dataset by 97.2% and introduce the way of adding new classes without exhaustive annotation. With the extended version of the dataset, we achieve 12.2% increase of mAP0.5 metric on average on the test data compared to the model trained on the original dataset.
In the cooperative vehicle infrastructure system (CVIS), perception data comes from multiple sources and modalities. Designing an optimal fusion mechanism to achieve more accurate perception is a key area of focus. In this paper, a multiple sources and multiple modalities self-supervised learning fusion perception framework (iIPPC-V2X) based on Invariant Linear Probabilistic Population Code(iIPPC) mechanism is proposed for vehicle-to-road cooperation. By simulating the neural integration mechanism of living organisms, the framework adaptively adjusts the weights of sensory information to achieve near-Bayesian optimal information integration. This paper also designs a vehicle infrastructure multiple sources and multiple modalities information fusion adjustment module to deal with the influence of external interference on the perception system, and evaluates the perception effect of the system based on mutual information theory. In addition, this paper proposes a context-aware self-supervised learning method based on large language model (LLM), which improves the robustness and adaptability of the perception model by augmenting data. The experimental results show that the proposed method significantly improves the accuracy and robustness of the cooperative vehicle infrastructure perception system in complex perception environments.
Egocentric 3D human pose estimation, which estimates an individual’s 3D pose from a camera attached to a part of their body, operates within a specialized camera domain to capture the entire body. This specialization makes it challenging to collect real-world training data due to the difficulty in acquiring diverse and accurately labeled data from the egocentric perspective. Consequently, most existing methods rely on synthetic data for training, which increases the mean joint error when applied to real-world images due to the domain gap. Some works have addressed this issue by generating pseudo-labels from synchronized real egocentric and exocentric images and using them for training. However, this approach is costly in terms of data collection, making it difficult to scale and apply to other camera setups. In this work, we propose a novel method that employs a diffusion model with ControlNet to generate real-world-like images, thereby reducing the domain gap. The proposed method relies on synthetic data, which is easy to acquire, and a small amount of text-captioned real-world data. This method easily applies to estimating egocentric 3D human poses across various camera setups. Experiments with two different camera setups demonstrated that models trained with images generated by the proposed method improve accuracy with real-world data. Specifically, the PA-MPJPE of the Mo2Cap2 model improved by 8.9
The capabilities of monocular depth estimation (MDE) models are limited by the availability of sufficient and diverse datasets. In the case of MDE models for autonomous driving, this issue is exacerbated by the linearity of the captured data trajectories. We propose a NeRF-based data augmentation pipeline to introduce synthetic data with more diverse viewing directions into training datasets and demonstrate the benefits of our approach to model performance and robustness. Our data augmentation pipeline, which we call NeRFmentation, trains NeRFs on each scene in a dataset, filters out subpar NeRFs based on relevant metrics, and uses them to generate synthetic RGB-D images captured from new viewing directions. In this work, we apply our technique in conjunction with three state-of-the-art MDE architectures on the popular autonomous driving dataset, KITTI, augmenting its training set of the Eigen split. We evaluate the resulting performance gain on the original test set, a separate popular driving dataset, and our own synthetic test set.
In the research field of analysis of people, generating precise full-body human motion from sparse tracking is a significant challenge. It is well known that diffusion techniques excel in generating high-quality two-dimensional (2D) visual content. However, when applied to human motion reconstruction, they might struggle to capture the inherent complexities of human motion, which is characterized by three-dimensional (3D) anatomical features and one-dimensional (1D) temporal dynamics. This heterogeneous structure between human motion and images can lead to accumulated errors at the joints, affecting the accuracy and smoothness of the generated motions. Building on this insight, we propose Human Anatomy Diffusion (HAD), a novel framework that integrates human anatomical features into the denoising process and excels in handling complex motions, accurately capturing body angles and balance, and showing enhanced alignment in motion prediction. HAD remarkably advanced the performance of motion reconstruction, notably enhancing smoothness by 81.29 https://niuzehai.github.io/had/ .
Object detectors suffer from reduced performance when they are utilized in real-world scenarios, where out-of-distribution (OOD) data exist together with in-distribution (ID) data. In a closed-world setting, the information present in data is not fully exploited, which results in unreliable models. The demand for models operating on both ID and OOD data is inevitable. We propose Class-Agnostic Point-to-Box Regressor (CA-PBR) for guiding the models to leverage unattended informative data in images without being restricted by class information. CA-PBR trains a point-to-box regressor in a class-agnostic fashion to generate credible object proposals. We additionally show that querying CA-PBR with a set of points obtained from a grid that is created on an image results in detection of novel instances with high diversity. Our experimental results show that utilizing CA-PBR as object proposal generator improves the detection of both known (ID) and novel (OOD) instances when trained on MS-COCO using only 10
The classification of leukocyte sub-types is essential for medical diagnostics and treatments. Advances in this field have been driven by the creation of novel Deep Learning (DL) architectures, whose progress is sometimes marginal or not even comparable due to the use of proprietary data sets or different setups/partitions of public data sets. This study presents a novel synthetic image data set designed for both training and benchmarking, providing a standardised platform to evaluate advancements in this field. The data set includes two versions of differing complexity: straightforward and challenging. Experiments with various DL models showed unexpectedly higher accuracy, precision, and recall on the more complex data set. These results highlight the importance of data set complexity in assessing the robustness and effectiveness of DL models for complex medical image analysis tasks.
We explore user-agnostic and user-specific mapping techniques for manipulating avatars using hand gestures in virtual reality (VR). User-agnostic mapping allows users to control various avatars based on common user agreements, while user-specific mapping adapts to individual preferences using few-shot adaptation. Both approaches use supervised learning with paired datasets of motion data. Our evaluation, including quantitative assessments and user studies, shows that both techniques offer finer control, require less physical effort, and provide higher user satisfaction compared to existing methods. However, only a few users preferred user-specific mapping, indicating that the benefits of personalized mapping may vary. Finally, we demonstrate our methods' ability to manipulate avatars with variable joint structures, surpassing current methods.
Road scene analysis methods usually classify driving events after they happen, providing valuable video evidence and coaching opportunities but not in-cabin alerts that could help prevent accidents. In-device models, on the other hand, try to identify dangerous events like frontal collision or lane departure beforehand and warn the driver before an accident happens. However, such systems typically employ object detectors plus hand-engineered rules, which fail to cover all possible corner cases. We propose a novel self-supervised end-to-end training approach. Our solution does not require object detection and is able to predict common dangerous situations like incoming forward collision or short following distance as well as corner cases like unusual driving behavior or potential collision with animals. By assuming a non-decreasing danger level in the instants just before the crash, we design a novel loss function to train our model without the need for hand-engineered rules. As a matter of fact, the proposed loss function is simple and generic enough that could be applied to any task with know start and end states. Our approach enables the usage of larger datasets for training by drastically reducing labeling effort while still maintaining competitive performance in the Car Crash Dataset (CCD) and Dashcam Accident Dataset (DAD), when compared to the literature. Finally, we show how our method generalizes to a variety of crash dynamics from the Dataset of Traffic Anomaly (DoTA), while still maintaining a contained inference time thanks to its streamlined approach.
Recent techniques like neural radiance fields (NeRFs) and 3D Gaussian splatting (3DGS) have led to significant improvements in novel view synthesis. Whereas the explicit scene representation of 3DGS in terms of Gaussians allows real-time rendering with state-of-the-art quality, this approach relies on the availability of many views to achieve a coherent scene representation. In this paper, we investigate the importance of accurate camera poses and demonstrate that this even allows for accurate scene representation based on 3D Gaussian Splatting in a sparse-view setting. For this purpose, we address accurate pose estimation by employing a measurement arm equipped with a camera, achieving precise camera-pose estimates with sub-millimeter accuracy. Based on a newly introduced dataset (Core dataset) with its accurate pose information, we demonstrate superior quality in terms of quality of rendered novel views in comparison to results achieved based on calibrations with Dust3R-based and COLMAP-based initializations of the 3D Gaussians. Thereby, our approach offers a reliable and effective solution to practical, sparse-view reconstruction for the preservation of cultural heritage artifacts, which is particularly relevant in applications like virtual museums and archaeology. Furthermore, we expect our Core dataset to serve as a reasonable benchmark, advancing the understanding and development of robust 3D reconstruction methods.
