Deep learning models can perform well when evaluated on images from the same distribution as the training set. However, applying small perturbations in the forms of noise, artifacts, occlusions, blurring, etc. to a model's input image and feeding the model with out-of-distribution (OOD) data can significantly drop the model's accuracy, making it not applicable to real-world scenarios. Data augmentation is one of the well-practiced methods to improve model robustness against OOD data; however, examining which augmentation type to choose and how it affects the OOD robustness remains understudied. There is a growing belief that augmenting datasets using data augmentations that improve a model's bias to shape-based features rather than texture-based features results in increased OOD robustness for Convolutional Neural Networks trained on the ImageNet-1K dataset. This is usually stated as “an increase in the model's shape bias results in an increase in its OOD robustness”. Based on this hypothesis, some works in the literature aim to find augmentations with higher effects on model shape bias and use those for data augmentation. By evaluating 39 types of data augmentations on a widely used OOD dataset, we demonstrate the impact of each data augmentation on the model's robustness to OOD data and further show that the mentioned hypothesis is not true; an increase in shape bias does not necessarily result in higher OOD robustness. By analyzing the results, we also find some biases in the ImageNet-1K dataset that can easily be reduced using proper data augmentation. Our evaluation results further show that there is not necessarily a trade-off between in-domain accuracy and OOD robustness, and choosing the proper augmentations can help increase both in-domain accuracy and OOD robustness simultaneously.
Modern approaches to scene graph generation still struggle with their performance, with even state of the art approaches hovering under a 15% mean recall on certain evaluation modes. This poor performance is partially a result of networks heavily relying and fixating on non-visual data, such as class statistics, instead of the pixel-level signals present in the images. We demonstrate this by examining the 'visual-ness' of visual relationship detection approaches. We first describe and implement a new Naive Bayes-based statistical baseline for scene graph generation. Most notably, this basic classifier does not utilize the image pixels, but relies on the properties of the bounding boxes (class labels, topological configuration, … etc.) to predict the relationship labels. We demonstrate that our classical machine learning approach, one as simple as a categorical Naive Bayes classifier, can perform relationship detection in a manner that achieves relatively competitive performance to that of modern scene graph generators. This is an alarming finding regarding scene graph generation that implies that visual data in images may not be utilized in modern visual relationship detection past the point of object detection. We finally discuss how more visual modern approaches to scene graph generation appear to remedy some of these shortcomings.
Target tracking is a classic problem in computer vision, with numerous applications in robotics. However, tracking targets underwater presents additional complications due to the six degrees of freedom nature of the problem and the challenging visual environment. In this paper, we address the problem of robotic underwater tracking of scuba divers by partitioning it into two parts: vision and control. We propose a new approach that exploits a highly-maneuverable underwater robot to perform experiments in open water, coupling sensing and control for improved performance. To evaluate the temporal stability of different tracking paradigms, we introduce a new metric, frame-to-frame vari-ance, which is better suited to assess the smoothness of detections from the vision side. We implement PID controllers for control and a spiral search algorithm for target recovery in case of a tracking failure. Our approach only uses observations in the image plane, eliminating the need for robot localization or camera calibration. Using a tracking-by-detection paradigm that combines YOLOv7 for target detection, a tuned filtering technique for temporal stability, and a spiral search algorithm for target recovery, we demonstrate promising performance for long-term tracking. We evaluate our proposed paradigm on the VDD-C dataset and deploy it on an underwater robot for several experiments in open water. Our outcomes show consistency with the ones in the initial studies, and the spiral search algorithm demonstrates promising performance for recapturing a target after a tracking failure. Our approach delivers promising performance for robust underwater tracking, achieving successful open-water tracking scenarios in the presence of strong water currents.
This paper analyzes the robustness of recent 3D shape descriptors to SO(3) rotations, something that is fundamental to shape modeling. Specifically, we formulate the task of rotated 3D object instance detection. To do so, we consider a database of 3D indoor scenes, where objects occur in different orientations. We benchmark different methods for feature extraction and classification in the context of this task. We systematically contrast different choices in a variety of experimental settings investigating the impact on the performance of different rotation distributions, different degrees of partial observations on the object, and the different levels of difficulty of negative pairs. Our study, on a synthetic dataset of 3D scenes where objects instances occur in different orientations, reveals that deep learning-based rotation invariant methods are effective for relatively easy settings with easy-to-distinguish pairs. However, their performance decreases significantly when the difference in rotations on the input pair is large, or when the degree of observation of input objects is reduced, or the difficulty level of input pair is increased. Finally, we connect feature encodings designed for rotation-invariant methods to 3D geometry that enable them to acquire the property of rotation invariance.
We propose a method that augments a simulated dataset using diffusion models to improve the performance of pedestrian detection in real-world data. The high cost of collecting and annotating data in the real-world has motivated the use of simulation platforms to create training datasets. While simulated data is inexpensive to collect and annotate, it unfortunately does not always closely match the distribution of real-world data, which is known as the sim2real gap. In this paper we propose a novel method of synthetic data creation meant to close the sim2real gap for the challenging pedestrian detection task. Our method uses a diffusion-based architecture to learn a real-world distribution which, once trained, is used to generate datasets. We mix this generated data with simulated data as a form of augmentation and show that training on a combination of generated and simulated data increases average precision by as much as 27.3% for pedestrian detection models in real-world data, compared against training on purely simulated data.
Predicting a binary mask for an object is more accurate but also more computationally expensive than a bounding box. Polygonal masks as developed in CenterPoly can be a good compromise. In this paper, we improve over CenterPoly by enhancing the classical regression L1 loss with a novel region-based loss and a novel order loss, as well as with a new training process for the vertices prediction head. Moreover, the previous methods that predict polygonal masks use different coordinate systems, but it is not clear if one is better than another, if we abstract the architecture requirement. We therefore investigate their impact on the prediction. We also use a new evaluation protocol with oracle predictions for the detection head, to further isolate the segmentation process and better compare the polygonal masks with binary masks. Our instance segmentation method is trained and tested with challenging datasets containing urban scenes, with a high density of road users. Experiments show, in particular, that using a combination of a regression loss and a region-based loss allows significant improvements on the Cityscapes and IDD test set compared to CenterPoly. Moreover the inference stage remains fast enough to reach real-time performance with an average of 0.045 s per frame for 2048x1024 images on a single RTX 2070 GPU. The code is available at: https://github.com/KatiaJDL/CenterPoly-v2.
Accurate 3D object detection in all weather conditions remains a key challenge to enable the widespread deployment of autonomous vehicles, as most work to date has been performed on clear weather data. In order to generalize to adverse weather conditions, supervised methods perform best if trained from scratch on all weather data instead of finetuning a model pretrained on clear weather data. Training from scratch on all data will eventually become computationally infeasible and expensive as datasets continue to grow and encompass the full extent of possible weather conditions. On the other hand, naive finetuning on data from a different weather domain can result in catastrophic forgetting of the previously learned domain. Inspired by the success of replay-based continual learning methods, we propose Gradient-based Maximally Interfered Retrieval (GMIR), a gradient based sampling strategy for replay. During finetuning, GMIR periodically retrieves samples from the previous domain dataset whose gradient vectors show maximal interference with the gradient vector of the current update. Our 3D object detection experiments on the SeeingThroughFog (STF) dataset [1] show that GMIR not only overcomes forgetting but also offers competitive performance compared to scratch training on all data with a 46.25% reduction in total training time.
Orchard tree inventory has been an essential step to obtain up-to-date information for effective tree treatments and crop insurance purposes. Inventorying trees is often performed manually through fieldwork surveys, which are generally time-consuming, costly, and subject to errors. Motivated by the latest advances in UAV imagery and deep learning, we propose a new framework for individual tree detection and health assessment. We adopt a divide-and-conquer approach to address the problem of orchard trees' health assessment in two stages. First, we build a tree detection model based on a hard negative mining strategy to improve object detection. In the second stage, we address the health classification problem using a new convolutional autoencoder architecture mainly designed to extract relevant features. The performed experiments demonstrate the robustness of the proposed framework for orchard tree health assessment from UAV images. In particular, our framework achieves an F1-score of 86.24% for tree detection and an overall accuracy of 98.06% for tree health assessment. Moreover, our work could be generalized for a wide range of UAV applications involving a detection/classification process.
The Distributed Deterministic Spiral Algorithm (DDSA) has shown great foraging efficiency in robot swarms. However, when the number of robots in the swarm increases, scalability becomes a significant bottleneck due to increased collisions among robots, making it challenging to deploy them in the search space (e.g., 20 robots). To address this issue, we propose an adaptive Multiple-Distributed Bidirectional Spiral Algorithm (MDBSA) that enhances scalability. Our proposed algorithm partitions the squared search arena into multiple identical squared regions and assigns robots to regions dynamically based on the number of regions. In each region, a bidirectional spiral search path is planned, and when a robot completes its search, it is assigned to either an unassigned region or a region with one robot. The two robots will then travel along the path from the starting and ending points of the spiral path. We evaluated the performance of robot swarms using the MDBSA algorithm in the ARGoS robot simulator. Our experimental results show that the proposed MDBSA algorithm outperforms DDSA. When robots deliver collected resources to regions instead of the center, it reduces collisions and significantly improves the scalability of the robot swarm. Our findings suggest that a multiple-distributed search strategy is an efficient solution for foraging robot swarms.
In standard captioning, the characteristics of the end-user whom we generate the caption for are ignored. This is mainly because more than often we do not have access to the entire spectrum of personality characteristics for our user. In other words, each user in test time can exhibit different traits to which we need to adapt our model. Therefore, we focus on generating personalized image captioning and formulate the problem as a few-shot learning setting. To the best of our knowledge, we are the first to study this problem and shed light on the challenges involved with this setting. Furthermore, we propose a MAML-based few-shot learner enabling the model to learn a new personality style from only a handful of annotated samples. Finally, we set up baselines for the problem and show that our proposed method is superior in performance when compared with baselines on the benchmark dataset. Ablation studies are conducted to investigate different design choices' effects on the model performance.
The success of transformer networks in the natural language processing and 2D vision domains has encouraged the adaptation of transformers for 3D computer vision tasks. However, majority of the existing approaches employ standard back-propagation (SBP) - which requires the storage of model activations on a forward pass for use during the backward pass - making their memory complexity linearly proportional to model depth, hence inefficient. Furthermore, most 3D point transformers use the classic QK(V) matrix multiplication design which comes with a memory bottleneck. To address these issues, we propose a memory-efficient point transformer that makes use of reversible functions and linearized self-attention to minimize SBP and transformer memory complexities, respectively. Experimental results on benchmark datasets (Toronto3D and CSPC) from different sensor platforms (aerial, and mobile backpack) show that our approach uses less than half the number of model parameters (compared to its SBP counterpart), take more than twice the input sequence, and use less than half the memory compared to majority of the traditional approach. Overall, the proposed RPT attained competitive performance compared to the state-of-the-art.
Neurophysiological studies suggest that neurons in the intermediate visual area V4 of the primate cortex encode a sparse representation of object shape. While there are metabolic arguments for such sparse representations, there are also potential advantages for inference. Here we explore whether sparse shape encoding can yield benefits for object instance segmentation. Specifically, we encode 2D object shape using a Distance Transform Map (DTM) and learn a sparse basis for this representation. To make use of this encoding, we design and train an instance segmentation head to estimate the sparse coefficients representing the shape of each object, and then recover the estimated shape from the zero-crossing level set of the corresponding DTM. Our novel SparseShape encoding approach produces fewer topological errors than the state of the art, yields competitive mask AP on the COCO benchmark and exhibits superior generalization performance on the Cityscapes instance segmentation task. These results suggest that the sparse shape encoding observed in primate cortex has computational advantages that can benefit computer vision instance segmentation systems.
The local shape of the occluding contour of an object is known to constrain the local shape of the object surface [1], however, these known constraints are qualitative. Here we posit here that in addition to these qualitative constraints, typical regularities of common objects and rules of projection induce dependencies that can be used to derive statistical estimates of quantitative solid shape from the occluding contour. To explore this conjecture, we partition the problem into two parts: 1) Estimation of the 3D rim from the 2D occluding contour, and 2) Estimation of the visible surface shape from the estimated 3D rim. We train and evaluate a number of statistical models on two distinct 3D object datasets and demonstrate that capturing these statistical regularities leads to better estimates of 3D shape than existing methods.
Recently, learning-based image signal processor (ISP) pipelines modeled using convolutional neural networks (CNNs) have been able to provide higher quality images over traditional model-based ISPs at the expense of significant memory, energy, and computation overhead. We propose an unrolled optimization network that models the ISP pipeline with considerably lower number of parameters and computation overhead. The unrolled optimization solves the image reconstruction problem of the ISP by leveraging both model-based and learning-based methods. In the proposed ISP model, the image formation operators namely, blur kernels and sensor sampling functions are formulated with learnable parameters such that the physical constraints are respected during conventional training. A CNN that is shared across the iterations of the unrolled model plays the role of the prior and performs denoising. An efficient tone mapper network is also utilized to further improve the quality of the resulting images. The entire pipeline is then trained in an end-to-end fashion using perceptual loss. The proposed ISP has over $\mathbf{34}\times$ fewer parameters in comparison to the state-of-the art deep ISPs.
Agitation is a key behavioural and psychological symptom exhibited by people with dementia. These behaviours can put the patient with dementia and others' health and safety at risk. Surveillance cameras installed in long-term care facilities provide an opportunity to monitor patients continuously and flag behaviours of risks, including agitation. However, agitation behaviours occur rarely and diversely, leading to very small training data. Therefore, an anomaly detection approach is more suitable for this problem. In this paper, we train three baseline spatio-temporal convolutional autoencoders (on raw video, skeletons and segmentation mask) on 21 hours of normal activities and tested it on 9 hours of labelled normal and agitation data collected from a real patient in a dementia unit. The deployment of anomaly detection-based classifiers is challenging in real-world due to the absence of a validation set to obtain an operating threshold to regulate true positive and false positive rates. We present a new approach to create a proxy validation set for unseen agitation events utilizing the outliers within normal activities, and trained two separate autoencoders on normal and outliers activities. Then, we present 11 empirical thresholding approaches (existing, adapted and new) using either only normal training data or the proxy validation set. Our results showed consistently across raw video, skeletons and segmentation masks input that incorporating a proxy validation set improved performance both in terms of geometric means and Matthew's correlation coefficient. This paper highlights the real-world deployment challenges and assessment of the limit of true positives or false positives that can be acceptable in a clinical care environment.
The ability to estimate the motion of objects is crucial for robotic agents to act and interact in their environments. This is particularly challenging for non-rigid objects, as they exhibit complex dynamics with large degrees of freedom. This work presents a methodology to learn deformation models of 3D elastic objects based on graph networks. Our approach exploits physical reasoning by introducing soft constraints into the loss function during training. We evaluate the proposed method in a simulation framework developed for robotic manipulation tasks. Results show that such a physics-informed model is able to predict more accurate object motions with less data than a purely data-driven based model.
Human action recognition in videos is an important task of computer vision that aims to automatically recognize and classify human actions in video sequences. However, accurately recognizing human actions can be challenging due to the complexity and variability of human motion and appearance. In this paper, we propose ActiViT, a novel approach for human action recognition in videos based on a Transformer architecture. Unlike existing methods that rely on convolutional or recurrent layers, our model is entirely based on the Transformer encoder, enabling us to leverage valuable information in action image patches features. We demonstrate that by dynamically selecting key patches guided by specific human poses, our model learns informative features useful for distinguishing between different actions. Our experimental results on real-world datasets convincingly demonstrate the effectiveness of our model and the importance of selecting discriminative key poses for action recognition.
This paper proposes a novel automatic thresholding method called Automatic Salience Thresholding (AST) for creating binary masks for detecting and removing satellite streaks in night sky imagery. Our approach utilizes a combination of Gaussian filtering, a salience-based thresholding technique, morphological filtering and line detection using Probabilistic Hough Transformations to identify the satellite trail in the image. We evaluated our method on diverse datasets of night sky images containing satellite trails in varying lighting conditions. Our results show that AST outperforms the compared methods regarding accuracy and speed. Our proposed AST method provides a reliable and efficient solution for detecting satellite trails while balancing selectivity and sensitivity.
Current deep neural networks (DNNs) have achieved remarkable accuracy in various downstream tasks. However, their training and fine-tuning are challenging due to several factors, such as limited computational resources, extended training and fine-tuning times, and over-fitting due to small datasets. To address these challenges, we propose a three-stage fast fine-tuning method that efficiently trains DNNs for edge devices. Our method combines curriculum learning and domain adaptation techniques to accelerate training while achieving comparable performance. First, we develop a data curriculum approach, which ranks the dataset according to difficulty and split it into the source domain (containing easy data) and the target domain (containing difficult data). Second, we adapt the pre-trained model from the source domain to the target domain using an unsupervised domain adaptation (UDA) method called Deep CORAL. Finally, we continue training the adapted model on the source domain with fewer epochs. Our method achieves high accuracy quickly on various modern neural network architectures and datasets such as CIFAR-10, CIFAR-100, and CINIC-10.
Object counting methods rely on density maps, which are heatmaps produced by placing Gaussian density over object locations. However, density maps are expensive to collect. To reduce the annotation burden, we propose a form of weak supervision that only requires object-based pairwise image rankings. These annotations can be collected rapidly with a single click per image pair and supply a weak signal for object quantity. However, a model learn to fit spurious patterns that satisfy the ranking constraint but do not rely on the objects. To encourage the network to solve the ranking constraints by localizing objects, we propose adversarial density map estimation. This method regularizes a ranking network's intermediate feature representation such that it corresponds to a plausible density map. We demonstrate the effectiveness of our method on several benchmark object counting datasets, and show results with a performance that approaches that of fully-supervised methods using data that can be collected with a fraction of the annotation burden. We release code for reproducibility: github.com/sfu-mial/Rank2Count
