We present a framework for end-to-end joint quantization of Vision Transformers trained on ImageNet for the purpose of image classification. Unlike prior post-training or block-wise reconstruction methods, we jointly optimize over the entire set of all layers and inter-block dependencies without any labeled data, scaling effectively with the number of samples and completing in just one hour on a single GPU for ViT-small. We achieve state-of-the-art W4A4 and W3A3 accuracies on ImageNet and, to the best of our knowledge, the first PTQ results that maintain strong accuracy on ViT, DeiT, and Swin-T models under extremely low-bit settings (W1.58A8), demonstrating the potential for efficient edge deployment. Furthermore, we introduce a data-free calibration strategy that synthesizes diverse, label-free samples using Stable Diffusion Turbo guided by learned multi-mode prompts. By encouraging diversity in both the learned prompt embeddings and the generated image features, our data-free approach achieves performance on par with real-data ImageNet calibration and surpasses simple text-prompt baselines such as "a photo of ".

We present Markov Map Nearest Neighbor V2 (M2N2V2), a novel and simple, yet effective approach which leverages depth guidance and attention maps for unsupervised and training-free point-prompt-based interactive segmentation. Following recent trends in supervised multimodal approaches, we carefully integrate depth as an additional modality to create novel depth-guided Markov-maps. Furthermore, we observe occasional segment size fluctuations in M2N2 during the interactive process, which can decrease the overall mIoU's. To mitigate this problem, we model the prompting as a sequential process and propose a novel adaptive score function which considers the previous segmentation and the current prompt point in order to prevent unreasonable segment size changes. Using Stable Diffusion 2 and Depth Anything V2 as backbones, we empirically show that our proposed M2N2V2 significantly improves the Number of Clicks (NoC) and mIoU compared to M2N2 in all datasets except those from the medical domain. Interestingly, our unsupervised approach achieves competitive results compared to supervised methods like SAM and SimpleClick in the more challenging DAVIS and HQSeg44K datasets in the NoC metric, reducing the gap between supervised and unsupervised methods.

Recent progress in interactive point prompt based Image Segmentation allows to significantly reduce the manual effort to obtain high quality semantic labels. State-of-the-art unsupervised methods use self-supervised pre-trained models to obtain pseudo-labels which are used in training a prompt-based segmentation model. In this paper, we propose a novel unsupervised and training-free approach based solely on the self-attention of Stable Diffusion. We interpret the self-attention tensor as a Markov transition operator, which enables us to iteratively construct a Markov chain. Pixel-wise counting of the required number of iterations along the Markov chain to reach a relative probability threshold yields a Markov-iteration-map, which we simply call a Markov-map. Compared to the raw attention maps, we show that our proposed Markov-map has less noise, sharper semantic boundaries and more uniform values within semantically similar regions. We integrate the Markov-map in a simple yet effective truncated nearest neighbor framework to obtain interactive point prompt based segmentation. Despite being training-free, we experimentally show that our approach yields excellent results in terms of Number of Clicks (NoC), even outperforming state-of-the-art training based unsupervised methods in most of the datasets. Code is available at https://github.com/mkarmann/m2n2.

A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g., LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities, i.e. using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.

Depth information is essential for on-board perception in autonomous driving and driver assistance. Monocular depth estimation (MDE) is very appealing since it allows for appearance and depth being on direct pixelwise correspondence without further calibration. Best MDE models are based on Convolutional Neural Networks (CNNs) trained in a supervised manner, i.e., assuming pixelwise ground truth (GT). Usually, this GT is acquired at training time through a calibrated multi-modal suite of sensors. However, also using only a monocular system at training time is cheaper and more scalable. This is possible by relying on structure-from-motion (SfM) principles to generate self-supervision. Nevertheless, problems of camouflaged objects, visibility changes, static-camera intervals, textureless areas, and scale ambiguity, diminish the usefulness of such self-supervision. In this paper, we perform monocular depth estimation by virtual-world supervision (MonoDEVS) and real-world SfM self-supervision. We compensate the SfM self-supervision limitations by leveraging virtual-world images with accurate semantic and depth supervision and addressing the virtual-to-real domain gap. Our MonoDEVSNet outperforms previous MDE CNNs trained on monocular and even stereo sequences.

Learnable keypoint detectors and descriptors are beginning to outperform classical hand-crafted feature extraction methods. Recent studies on self-supervised learning of visual representations have driven the increasing performance of learnable models based on deep networks. By leveraging traditional data augmentations and homography transformations, these networks learn to detect corners under adverse conditions such as extreme illumination changes. However, their generalization capabilities are limited to corner-like features detected a priori by classical methods or synthetically generated data. In this paper, we propose the Correspondence Network (CorrNet) that learns to detect repeatable keypoints and extract discriminative descriptions via unsupervised contrastive learning under spatial constraints. Our experiments show that CorrNet is not only able to detect low-level features such as corners, but also high-level features that represent similar objects present in a pair of input images through our proposed joint guided backpropagation of their latent space. Our approach obtains competitive results under viewpoint changes and achieves state-of-the-art performance under illumination changes.

Due to the difficulty of obtaining ground-truth labels, learning from virtual-world datasets is of great interest for real-world applications like semantic segmentation. From domain adaptation perspective, the key challenge is to learn domain-agnostic representation of the inputs in order to benefit from virtual data. In this paper, we propose a novel trident-like architecture that enforces a shared feature encoder to satisfy confrontational source and target constraints simultaneously, thus learning a domain-invariant feature space. Moreover, we also introduce a novel training pipeline enabling self-induced cross-domain data augmentation during the forward pass. This contributes to a further reduction of the domain gap. Combined with a selftraining process, we obtain state-of-the-art results on benchmark datasets (e.g. GTA5 or Synthia to Cityscapes adaptation). Code and pre-trained models are available at https://github.com/HMRC-AEL/TridentAdapt

Acquiring information about the road lane structure is a crucial step for autonomous navigation. To this end, several approaches tackle this task from different perspectives such as lane marking detection or semantic lane segmentation. However, to the best of our knowledge, there is yet no purely vision based end-to-end solution to answer the precise question: How to estimate the relative number or “ID” of the current driven lane within a multi-lane road or a highway? In this work, we propose a real-time, vision-only (i.e. monocular camera) solution to the problem based on a dual left-right convention. We interpret this task as a classification problem by limiting the maximum number of lane candidates to eight. Our approach is designed to meet low-complexity specifications and limited runtime requirements. It harnesses the temporal dimension inherent to the input sequences to improve upon high-complexity state-of-the-art models. We achieve more than 95% accuracy on a challenging test set with extreme conditions and different routes.

Acquiring information about the road lane structure is a crucial step for autonomous navigation. To this end, several approaches tackle this task from different perspectives such as lane marking detection or semantic lane segmentation. However, to the best of our knowledge, there is yet no purely vision based end-to-end solution to answer the precise question: How to estimate the relative number or "ID" of the current driven lane within a multi-lane road or a highway? In this work, we propose a real-time, vision-only (i.e. monocular camera) solution to the problem based on a dual left-right convention. We interpret this task as a classification problem by limiting the maximum number of lane candidates to eight. Our approach is designed to meet low-complexity specifications and limited runtime requirements. It harnesses the temporal dimension inherent to the input sequences to improve upon high-complexity state-of-the-art models. We achieve more than 95% accuracy on a challenging test set with extreme conditions and different routes.

Depth estimation provides essential information to perform autonomous driving and driver assistance. A promising line of work consists of introducing additional semantic information about the traffic scene when training CNNs for depth estimation. In practice, this means that the depth data used for CNN training is complemented with images having pixel-wise semantic labels where the same raw training data is associated with both types of ground truth, i.e., depth and semantic labels. The main contribution of this paper is to show that this hard constraint can be circumvented, i.e., that we can train CNNs for depth estimation by leveraging the depth and semantic information coming from heterogeneous datasets. In order to illustrate the benefits of our approach, we combine KITTI depth and Cityscapes semantic segmentation datasets, outperforming state-of-the-art results on monocular depth estimation.

In this paper, we focus on Exposure Fusion (EF) [ExposFusi2] for dynamic scenes. The task is to fuse multiple images obtained by exposure bracketing to create an image which comprises a high level of details. Typically, such images are not possible to obtain directly from a camera due to hardware limitations, e.g., a limited dynamic range of the sensor. A major problem of such tasks is that the images may not be spatially aligned due to scene motion or camera motion. It is known that the required alignment by image registration problems is ill-posed. In this case, the images to be aligned vary in their intensity range, which makes the problem even more difficult. To address the mentioned problems, we propose an end-to-end \emph{Convolutional Neural Network} (CNN) based approach to learn to estimate exposure fusion from $2$ and $3$ Low Dynamic Range (LDR) images depicting different scene contents. To the best of our knowledge, no efficient and robust CNN-based end-to-end approach can be found in the literature for this kind of problem. The idea is to create a dataset with perfectly aligned LDR images to obtain ground-truth exposure fusion images. At the same time, we obtain additional LDR images with some motion, having the same exposure fusion ground-truth as the perfectly aligned LDR images. This way, we can train an end-to-end CNN having misaligned LDR input images, but with a proper ground truth exposure fusion image. We propose a specific CNN-architecture to solve this problem. In various experiments, we show that the proposed approach yields excellent results.

Depth estimation provides essential information to perform autonomous driving and driver assistance. In particluar, monocular depth estimation is interesting from a practical point of view, since using a single camera is cheaper than many other options and avoids the need for continuous calibration strategies as required by stereo-vision approaches. State-of-the-art methods for monocular depth estimation are based on Convolutional Neural Networks (CNNs). A promising line of work consists of introducing additional semantic information about the traffic scene when training CNNs for depth estimation. In practice, this means that the depth data used for CNN training is complemented with images having pixelwise semantic labels, which usually are difficult to annotate (e.g. crowded urban images). Moreover, so far it is common practice to assume that the same raw training data is associated with both types of ground truth, i.e., depth and semantic labels. The main contribution of this paper is to show that this hard constraint can be circumvented, i.e., that we can train CNNs for depth estimation by leveraging the depth and semantic information coming from heterogeneous datasets. In order to illustrate the benefits of our approach, we combine KITTI depth and Cityscapes semantic segmentation datasets, outperforming stateof-the-art results on monocular depth estimation.

Despite being an essential prerequisite at the basis of many applications ranging from surveillance to computational photography, the problem of initial background estimation seems to be marginally investigated. In this paper, we present a reliable CNN-based solution to estimate the initial background (BG) of a scene, given not necessarily a whole sequence but just a small set of frames containing foreground objects (FG). The proposed solution is based on a convolutional neural network (CNN) which is trained to estimate BG patches followed by an aggregation/post-processing step of these estimates to form the final BG image. The accuracy of our approach is evaluated visually and numerically using different metrics on the proposed sequences by the scene background modeling contest 2016 (SBMC2016). It demonstrates robustness against very challenging scenarios under extreme conditions such as very short or long sequences, dynamic BG, illumination changes and intermittent object motion. As most deep learning solutions, our approach achieves promising results.

High Dynamic Range Imaging (HDRI) and Exposure Fusion (EF) are methods of choice to computationally extend the dynamic range of images depicting real world scenes. Unfortunately, those methods are still prone to certain artifacts. Among others, the so-called Ghost Effect is the most critical HDR limitation when it comes to dealing with motion (camera or scene motion) in input Low Dynamic Range (LDR) images. This problem becomes more challenging when the input LDR image stack contains only a couple of images with large color differences, which is the case in the mobile phone domain. To address this issue, a de-ghosting step is required to preserve the quality of the final HDR images. In this paper, we propose a robust de-ghosting approach based on the detection and the elimination of the motion induced effects on the final HDR images. The proposed method performs efficiently in all cases even on scenarios where only two differently exposed images with large illumination variations are available as input. Compared to the state-of-the-art, our results exhibit significant visual improvement and artifact reduction. Furthermore, our approach has low computational cost and complexity, which enables an efficient implementation especially for mobile phone-related applications.

Color mapping is a fundamental task for many important computer vision applications such as High Dynamic Range Imaging (HDRI), Stereo Matching, Camera Calibration and various other tasks. Typically, the task of color mapping is to transfer the colors of an image to a reference distribution. For example, this way, it is possible to simulate different camera exposures using a single image, e.g., by transforming a dark image to a brighter image showing the same scene. Most approaches for color mapping are local in the sense that they just apply a pixel-wise (local) mapping to generate the color mapped image. In this paper, we empirically show that this approach yields sub-optimal results and we propose a non-local mapping based on learned features directly from the image-texture, using a Convolutional Neural Network. This way, we learn to generate an image which would have been captured by a certain factor of the actual exposure time. We demonstrate our method using various applications in the HDR domain and compare our results against other state-of-the-art methods where we obtain excellent results, both visually as well as numerically.

We investigate the problem of depth map estimation of a scene from 4D light-field data. Unlike prior work, we process all the input images (or sub-images) of the light-field. Indeed, for each point of the scene, different depth candidates are estimated by considering all possible 3D light-field cuts instead of only 2, i.e. the horizontal and vertical one. Then, the optimal candidates are chosen by finding the labeling with minimum energy. The main motivation of our approach is that by processing multiple cuts of the light-field, the matching ambiguities are reduced. Our meta-method is of broad interest as it can be applied and enhance the performance of various state-of-the-art light-field depth estimation techniques. For the sake of demonstration we apply it on the work from Wanner and Goldluecke [19] and significantly improve their results.

Histogram Matching (HM) is a well known technique for matching the colors of a source image to those of a reference image. A major application is High Dynamic Range Imaging, where images with different exposures have to be color matched. HM works best when the images are aligned and the color difference is small. However, due to camera or object motion, the images may not be aligned, which can result in color artifacts. Furthermore, HM typically involves brightening dark regions of the source image, which increases the image noise. As a result, there is a need for detecting and correcting color artifacts and increased image noise. In this paper we present a novel post-processing method for improving the quality of histogram matched images using statistical properties of the reference image. The proposed method is based on ensembles of neighboring pixels. We learn the statistics of such ensembles within the reference image (target image), which exhibits content difference in comparison to the source image. Based on these statistics, we detect and reconstruct faulty intensities, by following a naive Bayes approach. In addition to its relative easiness of implementation, experimental results show considerable subjective (visual) as well as objective (PSNR) improvements over the original HM. Moreover, promising results are obtained on comparisons of our method with Non-Local-Means denoising based post-processing, which is a state-of-the-art denoising method.

Most of the work related to image denoising is based on artificial noise of stationary Gaussian distribution (synthetically added to the image in arrears). However, this choice is only a rough approximation of the real noise distribution. That is why, various research works were recently focusing on how to perfectly model the real noise inherent in captured images. In this paper, we model this distribution as non-stationary conditional Gaussian, where the standard deviation is depending on the pixel intensity. Experimental results show that this assumption models the real image noise more accurately (specifically when it comes to its adaptive removal). For that, we developed an extended version of BM3D called NBM3D. It is suitable for adaptive non-stationary noise removal in natural captured images where the original BM3D performance is relatively limited. To accurately verify the denoising performance under real case scenarios, we compute a "noise-free" ground truth image as the average of a sequence of images captured on a static scene by a non-moving monocular camera. Then, we estimate the conditional Gaussian distribution. The resulting model is used for the execution of NBM3D. The comparison between our method and the state-of-the-art BM3D is done by comparing the denoised images using both approaches against the ground truths. Results on three different cameras and 15 sequences with varying lighting conditions show that the proposed NBM3D provides consistent denoising improvements compared to state-of-the-art in terms of real noise removal.

In this paper we present a method for registering a pair of differently exposed Low Dynamic Range (LDR) images for the purpose of rendering a High Dynamic Range (HDR) image. In general, the images are captured from a moving camera and/or contain moving objects. Therefore, proper registration is required to enable HDR rendering. However, even for equally exposed images, registration is an ill posed problem where errors are expected for a wide range of image pairs. The problem only becomes more challenging for a pair of differently exposed images. We propose an adaptive registration error detection and correction method to address this issue. By combining Optical Flow with the proposed correction method, we achieve state-of-the art results as shown in numerous experiments. The proposed method is simple and has low-complexity, hence allowing for an easy and efficient implementation.

Differential Evolution (DE) is a widely used successful evolutionary algorithm (EA) based on a population of individuals, which is especially well suited to solve problems that have non-linear, multimodal cost functions. However, for a given population, the set of possible new populations is finite and a true subset of the cost function domain. Furthermore, the update formula of DE does not use any information about the fitness of the population. This paper presents a novel extension of DE called Randomized and Rank-based Differential Evolution (R2DE) and its self-adaptive version SAR2DE to improve robustness and global convergence speed on multimodal problems by introducing two multiplicative terms in the DE update formula. The first term is based on a random variate of a Cauchy distribution, which leads to a randomization. The second term is based on ranking of individuals, so that R2DE exploits additional information provided by the population fitness. In extensive experiments conducted with a wide range of complexity settings, we show that the proposed heuristics lead to an overall improvement in robustness and speed of convergence compared to several global optimization techniques, including DE, Opposition based Differential Evolution (ODE), DE with Random Scale Factor (DERSF) and the self-adaptive Cauchy distribution based DE (NSDE).