Many approaches have been proposed to use diffusion models to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large datasets, often with noisy annotations, and it remains an open question to which extent these models contribute to downstream classification performance. In particular, it remains unclear if they generalize enough to improve over directly using the additional data of their pre-training process for augmentation. We systematically evaluate a range of existing methods to generate images from diffusion models and study new extensions to assess their benefit for data augmentation. Personalizing diffusion models towards the target data outperforms simpler prompting strategies. However, using the pre-training data of the diffusion model alone, via a simple nearest-neighbor retrieval procedure, leads to even stronger downstream performance. Our study explores the potential of diffusion models in generating new training data, and surprisingly finds that these sophisticated models are not yet able to beat a simple and strong image retrieval baseline on simple downstream vision tasks.

Different urban microscale models exist to model street-level mean radiation temperature (Tmrt). However, these models are computationally expensive, albeit to varying degrees. We present a computational shortcut using a convolutional encoder-decoder network (U-Net) to predict pedestrian level (1.1 m a.g.l.) Tmrt at a building-resolved scale (1 x 1 m). SOLWEIG is used to create spatial training data for 68 days at hourly resolution in the city of Freiburg, Germany. Validation of the model was carried out in two steps: First, SOLWEIG (and U-Net) were validated against Tmrt point measurements. Second, U-Net was validated against SOLWEIG on 6 areas and 12 days resulting in a MAE of 2.4 K. The U-Net is 22 times faster than SOLWEIG, and thus able to emulate a micrometeorological physical model with computational superiority. As a demon-stration case, U-Net is applied to model Tmrt for the urbanized area of Freiburg for two complete 30-year periods (1961-1990, 1991-2020) driven by hourly ERA5-Land reanalysis data. Sum-mertime daily maximum Tmrt increased on average by 2.5 K, whereas summertime daily maximum air temperature increased by only 1.5 K. Maximum Tmrt increase is stronger on non -tree covered paved areas (2.8 K) than on tree covered grassy areas (1.8 K).

We present a novel method to model spatial maps of mean radiant temperature (Tmrt) in complex urban areas using a special type of fully convolutional networks - U-Net - for image to image processing. Tmrt is one of the driving factors of daytime human thermal comfort and underlies great spatial and temporal variabilities, especially in complex urban areas. Various micro scale (building-resolving) models exist to model Tmrt in urban settings. However, these models are computational expensive, albeit to varying degrees. This means, study area and time might be limited depending on spatial and temporal resolution. While this is sufficient for case studies where micro-level processes are modelled for different neighbourhoods in limited time periods, accurate calculations over a long time period are not possible (e.g. downscaling global climate projections). To overcome these computational drawbacks of physical models, we present a U-net approach for modelling Tmrt in complex urban areas. U-Nets are special types of encoder-decoder networks and allow precise image to image processing. In this study, Tmrt (at 1.1 m a.g.l.) is modelled by SOLWEIG model for 62 areas (500 x 500 m2) and on 54 days for the city of Freiburg, Germany. Training data is sampled randomly after clustering. The spatial and temporal input of SOLWEIG are in turn used as input features for the U-Net. The U-Net is trained on 56 areas and on 45 days and tested on the remaining areas and days. In addition, data from a Tmrt measurement campaign is used to validate SOLWEIG and U-Net model output. Results indicate that the proposed U-Net approach is capable to provide Tmrt in complex urban areas sufficiently. A correlation of > 0.9 and a MAE of 1.53°C between SOLWEIG and the U-Net is observed. Results show a higher MAE during day than night, which can be partly explained by the difference of absolute Tmrt values at day and night, but also by more complex prediction conditions during day: cloud cover and thus varying radiation, but also low sun angle in the morning / evening. In addition, computing times for Tmrt map predictions are significantly faster than physical models.

Predicting the future trajectory of a moving agent can be easy when the past trajectory continues smoothly but is challenging when complex interactions with other agents are involved. Recent deep learning approaches for trajectory prediction show promising performance and partially attribute this to successful reasoning about agent-agent interactions. However, it remains unclear which features such black-box models actually learn to use for making predictions. This paper proposes a procedure that quantifies the contributions of different cues to model performance based on a variant of Shapley values. Applying this procedure to state-of-the-art trajectory prediction methods on standard benchmark datasets shows that they are, in fact, unable to reason about interactions. Instead, the past trajectory of the target is the only feature used for predicting its future. For a task with richer social interaction patterns, on the other hand, the tested models do pick up such interactions to a certain extent, as quantified by our feature attribution method. We discuss the limits of the proposed method and its links to causality

Predicting the states of dynamic traffic actors into the future is important for autonomous systems to operate safely and efficiently. Remarkably, the most critical scenarios are much less frequent and more complex than the uncritical ones. Therefore, uncritical cases dominate the prediction. In this paper, we address specifically the challenging scenarios at the long tail of the dataset distribution. Our analysis shows that the common losses tend to place challenging cases sub-optimally in the embedding space. As a consequence, we propose to supplement the usual loss with a loss that places challenging cases closer to each other This triggers sharing information among challenging cases and learning specific predictive features. We show on four public datasets that this leads to improved performance on the challenging scenarios while the overall performance stays stable. The approach is agnostic w.r.t. the used network architecture, input modality or viewpoint, and can be integrated into existing solutions easily. Code is available at github.

Medical image datasets are hard to collect, expensive to label, and often highly imbalanced. The last issue is underestimated, as typical average metrics hardly reveal that the often very important minority classes have a very low accuracy. In this paper, we address this problem by a feature embedding that balances the classes using contrastive learning as an alternative to the common cross-entropy loss. The approach is largely orthogonal to existing sampling methods and can be easily combined with those. We show on the challenging ISIC2018 and APTOS2019 datasets that the approach improves especially the accuracy of minority classes without negatively affecting the majority ones.

In this paper, we investigate the problem of anticipating future dynamics, particularly the future location of other vehicles and pedestrians, in the view of a moving vehicle. We approach two fundamental challenges: (1) the partial visibility due to the egocentric view with a single RGB camera and considerable field-of-view change due to the egomotion of the vehicle; (2) the multimodality of the distribution of future states. In contrast to many previous works, we do not assume structural knowledge from maps. We rather estimate a reachability prior for certain classes of objects from the semantic map of the present image and propagate it into the future using the planned egomotion. Experiments show that the reachability prior combined with multi-hypotheses learning improves multimodal prediction of the future location of tracked objects and, for the first time, the emergence of new objects. We also demonstrate promising zero-shot transfer to unseen datasets.

Future prediction is a fundamental principle of intelligence that helps plan actions and avoid possible dangers. As the future is uncertain to a large extent, modeling the uncertainty and multimodality of the future states is of great relevance. Existing approaches are rather limited in this regard and mostly yield a single hypothesis of the future or, at the best, strongly constrained mixture components that suffer from instabilities in training and mode collapse. In this work, we present an approach that involves the prediction of several samples of the future with a winner-takes-all loss and iterative grouping of samples to multiple modes. Moreover, we discuss how to evaluate predicted multimodal distributions, including the common real scenario, where only a single sample from the ground-truth distribution is available for evaluation. We show on synthetic and real data that the proposed approach triggers good estimates of multimodal distributions and avoids mode collapse.

Optical flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology. In this paper, we make such networks estimate their local uncertainty about the correctness of their prediction, which is vital information when building decisions on top of the estimations. For the first time we compare several strategies and techniques to estimate uncertainty in a large-scale computer vision task like optical flow estimation. Moreover, we introduce a new network architecture utilizing the Winner-Takes-All loss and show that this can provide complementary hypotheses and uncertainty estimates efficiently with a single forward pass and without the need for sampling or ensembles. Finally, we demonstrate the quality of the different uncertainty estimates, which is clearly above previous confidence measures on optical flow and allows for interactive frame rates.

Recent work has shown that convolutional neural networks (CNNs) can be used to estimate optical flow with high quality and fast runtime. This makes them preferable for real-world applications. However, such networks require very large training datasets. Engineering the training data is difficult and/or laborious. This paper shows how to augment a network trained on an existing synthetic dataset with large amounts of additional unlabelled data. In particular, we introduce a selection mechanism to assemble from multiple estimates a joint optical flow field, which outperforms that of all input methods. The latter can be used as proxy-ground-truth to train a network on real-world data and to adapt it to specific domains of interest. Our experimental results show that the performance of networks improves considerably, both, in cross-domain and in domain-specific scenarios. As a consequence, we obtain state-of-the-art results on the KITTI benchmarks.

Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.