In this paper, we propose an efficient, fast, and versatile distillation method to accelerate the generation of pre-trained diffusion models. The method reaches state-of-the-art performances in terms of FID and CLIP-Score for few steps image generation on the COCO2014 and COCO2017 datasets, while requiring only several GPU hours of training and fewer trainable parameters than existing methods. In addition to its efficiency, the versatility of the method is also exposed across several tasks such as *text-to-image*, *inpainting*, *face-swapping*, *super-resolution* and using different backbones such as UNet-based denoisers (SD1.5, SDXL), DiT (Pixart) and MMDiT (SD3), as well as adapters. In all cases, the method allowed to reduce drastically the number of sampling steps while maintaining very high-quality image generation.

Realistic shadow generation is a critical component for high-quality image compositing and visual effects, yet existing methods suffer from certain limitations: Physics-based approaches require a 3D scene geometry, which is often unavailable, while learning-based techniques struggle with control and visual artifacts. We introduce a novel method for fast, controllable, and background-free shadow generation for 2D object images. We create a large synthetic dataset using a 3D rendering engine to train a diffusion model for controllable shadow generation, generating shadow maps for diverse light source parameters. Through extensive ablation studies, we find that rectified flow objective achieves high-quality results with just a single sampling step enabling real-time applications. Furthermore, our experiments demonstrate that the model generalizes well to real-world images. To facilitate further research in evaluating quality and controllability in shadow generation, we release a new public benchmark containing a diverse set of object images and shadow maps in various settings. The project page is available at https://gojasper.github.io/controllable-shadow-generation-project/

The domain adaptation of satellite images has recently gained increasing attention to overcome the limited generalization abilities of machine learning models when segmenting large-scale satellite images. Most of the existing approaches seek for adapting the model from one domain to another. However, such single-source and single-target setting prevents the methods from being scalable solutions since, nowadays, multiple sources and target domains having different data distributions are usually available. Besides, the continuous proliferation of satellite images necessitates the classifiers to adapt to continuously increasing data. We propose a novel approach, coined DAugNet, for unsupervised, multisource, multitarget, and life-long domain adaptation of satellite images. It consists of a classifier and a data augmentor. The data augmentor, which is a shallow network, is able to perform style transfer between multiple satellite images in an unsupervised manner, even when new data are added over time. In each training iteration, it provides the classifier with diversified data, which makes the classifier robust to large data distribution difference between the domains. Our extensive experiments prove that DAugNet significantly better generalizes to new geographic locations than the existing approaches.

Chapter 7 Deep Domain Adaptation in Earth Observation Benjamin Kellenberger, Benjamin KellenbergerSearch for more papers by this authorOnur Tasar, Onur TasarSearch for more papers by this authorBharath Bhushan Damodaran, Bharath Bhushan DamodaranSearch for more papers by this authorNicolas Courty, Nicolas CourtySearch for more papers by this authorDevis Tuia, Devis TuiaSearch for more papers by this author Benjamin Kellenberger, Benjamin KellenbergerSearch for more papers by this authorOnur Tasar, Onur TasarSearch for more papers by this authorBharath Bhushan Damodaran, Bharath Bhushan DamodaranSearch for more papers by this authorNicolas Courty, Nicolas CourtySearch for more papers by this authorDevis Tuia, Devis TuiaSearch for more papers by this author Book Editor(s):Gustau Camps-Valls, Gustau Camps-Valls Universitat de València, SpainSearch for more papers by this authorDevis Tuia, Devis Tuia EPFL, SwitzerlandSearch for more papers by this authorXiao Xiang Zhu, Xiao Xiang Zhu German Aerospace Center and Technical University of Munich, GermanySearch for more papers by this authorMarkus Reichstein, Markus Reichstein Max Planck Institute, GermanySearch for more papers by this author First published: 20 August 2021 https://doi.org/10.1002/9781119646181.ch7 AboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Summary When applied to new datasets, acquired at different time moments, with different sensors or under different acquisition conditions, deep learning models might fail spectacularly. This is because they have learned from the data distribution observed during training and, as such, do not generalize out of that domain naturally. This chapter introduces methodologies designed to tackle this problem and provide deep learning models able to adapt to new data distributions, i.e. domain adaptation. Domain adaptation works by either adapting the representation to the new data distribution, modifying the inputs or performing smart sampling. But independently of the strategy, they lead to updated models, able to process effectively the new data without needing observation from it (or a very limited amount). Deep Learning for the Earth Sciences: A Comprehensive Approach to Remote Sensing, Climate Science, and Geosciences RelatedInformation

Faults form dense, complex multi‐scale networks generally featuring a master fault and myriads of smaller‐scale faults and fractures off its trace, often referred to as damage. Quantification of the architecture of these complex networks is critical to understanding fault and earthquake mechanics. Commonly, faults are mapped manually in the field or from optical images and topographic data through the recognition of the specific curvilinear traces they form at the ground surface. However, manual mapping is time‐consuming, which limits our capacity to produce complete representations and measurements of the fault networks. To overcome this problem, we have adopted a machine learning approach, namely a U‐Net Convolutional Neural Network (CNN), to automate the identification and mapping of fractures and faults in optical images and topographic data. Intentionally, we trained the CNN with a moderate amount of manually created fracture and fault maps of low resolution and basic quality, extracted from one type of optical images (standard camera photographs of the ground surface). Based on a number of performance tests, we select the best performing model, M Ref , and demonstrate its capacity to predict fractures and faults accurately in image data of various types and resolutions (ground photographs, drone and satellite images and topographic data). M Ref exhibits good generalization capacities, making it a viable tool for fast and accurate mapping of fracture and fault networks in image and topographic data. The M Ref model can thus be used to analyze fault organization, geometry, and statistics at various scales, key information to understand fault and earthquake mechanics.

Domain adaptation for semantic segmentation has recently been actively studied to increase the generalization capabilities of deep learning models. The vast majority of the domain adaptation methods tackle single-source case, where the model trained on a single source domain is adapted to a target domain. However, these methods have limited practical real world applications, since usually one has multiple source domains with different data distributions. In this work, we deal with the multi-source domain adaptation problem. Our method, namely StandardGAN, standardizes each source and target domains so that all the data have similar data distributions. We then use the standardized source domains to train a classifier and segment the standardized target domain. We conduct extensive experiments on two remote sensing data sets, in which the first one consists of multiple cities from a single country, and the other one contains multiple cities from different countries. Our experimental results show that the standardized data generated by StandardGAN allow the classifiers to generate significantly better segmentation.

Due to the various reasons, such as atmospheric effects and differences in acquisition, it is often the case that there exists a large difference between the spectral bands of satellite images collected from different geographic locations. The large shift between the spectral distributions of training and test data causes the current state-of-the-art supervised learning approaches to output unsatisfactory maps. We present a novel semantic segmentation framework that is robust to such a shift. The key component of the proposed framework is color mapping generative adversarial networks (ColorMapGANs) that can generate fake training images that are semantically exactly the same as training images, but whose spectral distribution is similar to the distribution of the test images. We then use the fake images and the ground truth for the training images to fine-tune the already trained classifier. Contrary to the existing generative adversarial networks (GANs), the generator in ColorMapGAN does not have any convolutional or pooling layers. It learns to transform the colors of the training data to the colors of the test data by performing only one elementwise matrix multiplication and one matrix-addition operation. Due to the architecturally simple but powerful design of ColorMapGAN, the proposed framework outperforms the existing approaches with a large margin in terms of both accuracy and computational complexity.

Although convolutional neural networks have been proven to be an effective tool to generate high quality maps from remote sensing images, their performance significantly deteriorates when there exists a large domain shift between training and test data. To address this issue, we propose a new data augmentation approach that transfers the style of test data to training data using generative adversarial networks. Our semantic segmentation framework consists in first training a U-net from the real training data and then fine-tuning it on the test stylized fake training data generated by the proposed approach. Our experimental results prove that our framework outperforms the existing domain adaptation methods.

In this work, we propose a novel multi-task framework, to learn satellite image pansharpening and segmentation jointly. Our framework is based on the encoder-decoder architecture, where both tasks share the same encoder but each one has its own decoder. We compare our framework against single-task models with different architectures. Results show that our framework outperforms all other approaches in both tasks.

In dense labeling problem, the major drawback of the convolutional neural networks is their inability to learn new classes without affecting performance for the old classes on the data, having no annotations for the previous classes. In this work, we address the issue of adding new classes continually to the already trained network from a stream of data. Our approach comprises two main components: adaptation and remembering. For adaptation, we keep a clone of the previously trained network, which serves as a memory for the old classes in absence of their annotations on the new data. The updated network learns new as well as old classes on the current data using output of the memory network and the new ground-truth. For remembering, we store a little portion of the previous data, from which we systematically feed samples to the updated network during training. Our results prove that segmentation capabilities for the new classes can be added to the already trained network without catastrophically forgetting the previously learned information.

In spite of remarkable success of the convolutional neural networks on semantic segmentation, they suffer from catastrophic forgetting: a significant performance drop for the already learned classes when new classes are added on the data having no annotations for the old classes. We propose an incremental learning methodology, enabling to learn segmenting new classes without hindering dense labeling abilities for the previous classes, although the entire previous data are not accessible. The key points of the proposed approach are adapting the network to learn new as well as old classes on the new training data, and allowing it to remember the previously learned information for the old classes. For adaptation, we keep a frozen copy of the previously trained network, which is used as a memory for the updated network in the absence of annotations for the former classes. The updated network minimizes a loss function, which balances the discrepancy between outputs for the previous classes from the memory and updated networks, and the misclassification rate between outputs for the new classes from the updated network and the new ground-truth. For remembering, we either regularly feed samples from the stored, little fraction of the previous data or use the memory network, depending on whether the new data are collected from completely different geographic areas or from the same city. Our experimental results prove that it is possible to add new classes to the network, while maintaining its performance for the previous classes, despite the whole previous training data are not available.

The Geographic Object-Based Image Analysis (GEOBIA) paradigm relies strongly on the segmentation concept, i.e., partitioning of an image into regions or objects that are then further analyzed. Segmentation is a critical step, for which a wide range of methods, parameters and input data are available. To reduce the sensitivity of the GEOBIA process to the segmentation step, here we consider that a set of segmentation maps can be derived from remote sensing data. Inspired by the ensemble paradigm that combines multiple weak classifiers to build a strong one, we propose a novel framework for combining multiple segmentation maps. The combination leads to a fine-grained partition of segments (super-pixels) that is built by intersecting individual input partitions, and each segment is assigned a segmentation confidence score that relates directly to the local consensus between the different segmentation maps. Furthermore, each input segmentation can be assigned some local or global quality score based on expert assessment or automatic analysis. These scores are then taken into account when computing the confidence map that results from the combination of the segmentation processes. This means the process is less affected by incorrect segmentation inputs either at the local scale of a region, or at the global scale of a map. In contrast to related works, the proposed framework is fully generic and does not rely on specific input data to drive the combination process. We assess its relevance through experiments conducted on ISPRS 2D Semantic Labeling. Results show that the confidence map provides valuable information that can be produced when combining segmentations, and fusion at the object level is competitive w.r.t. fusion at the pixel or decision level.

The Object-Based Image Analysis (OBIA) paradigm strongly relies on the concept of segmentation, i.e. partitioning of an image into regions or objects that are further analyzed (e.g., described and classified). This segmentation step is thus critical, while remaining a challenging issue since there is no (and probably will never be) perfect segmentation technique yet. Indeed, various segmentation criteria as well as input data can be exploited to derive a unique segmentation that could serve for further analysis. In order to alleviate the sensitivity of the OBIA process to the segmentation step, we consider here that a set of segmentation maps can be derived from remote sensing data. Such various segmentations can be obtained through several segmentation algorithms, multiple parameter settings, or even complementary remotely-sensed observations of a given scene (possibly coming at different spatial scales). Inspired from the boosting paradigm, we thus propose a novel framework for combining multiple segmentations maps. The combination leads to a fine-grained partition of segments (superpixels) that is built by intersecting individual input partitions. More importantly, each segment is assigned a segmentation confidence score that directly relates to the correlation of the different individual segmentation inputs regarding this segment. Furthermore, each input segmentation can be assigned some local or global quality score based on expert assessment or automatic analysis. These scores are then taken into account while computing the confidence map resulting from the combination of segmentations process. It helps the process not being too much affected by incorrect segmentation inputs either at the local scale of a region or at the global scale of a map. Some experiments conducted on the ISPRS 2D Semantic Labeling dataset (made of VHR color/multispectral optical image and DSM) have been conducted to assess the relevance of the proposed framework. Results show that confidence map is a valuable information that can be produced when combining segmentations. Furthermore, the segmentation map is used to perform object-based supervised classification based on random forests. To do so, we consider several strategies in order to exploit the confidence map, either through direct thresholding or as an intermediate image on which a final segmentation is applied. A comparison with individual segmentations shows our framework is able to effectively combine various segmentation maps in order to improve the subsequent classification. From a user perspective, this approach is also able to provide a confidence map related to the geometry of the classified objects, in addition to the probability-based uncertainty map related to their semantics.

One of the most popular and challenging tasks in remote sensing applications is the generation of digitized representations of Earth's objects from satellite raster image data. A common approach to tackle this challenge is a two-step method that first involves performing a pixel-wise classification of the raster data, then vectorizing the obtained classification map. We propose a novel approach, which recasts the polygonization problem as a mesh-based approximation of the input classification map, where binary labels are assigned to the mesh triangles to represent the building class. A dense initial mesh is decimated and optimized using local edge and vertex-based operators in order to minimize an objective function that models a balance between fidelity to the classification map in l1 norm sense, right angle regularity for polygonized buildings, and final mesh complexity. Experiments show that adding the right angle objective yields better representations quantitatively and qualitatively than previous work and commonly used polygon generalization methods in remote sensing literature for similar number of vertices.

One of the most popular and challenging tasks in remote sensing applications is the generation of digitized representations of Earth's objects from satellite raster image data. A common approach to tackle this challenge is a two-step method that first involves performing a pixel-wise classification of the raster data, then vectorizing the obtained classification map. We propose a novel approach, which recasts the polygonization problem as a mesh-based approximation of the input classification map, where binary labels are assigned to the mesh triangles to represent the building class. A dense initial mesh is decimated and optimized using local edge and vertex-based operators in order to minimize an objective function that models a balance between fidelity to the classification map in l(1) norm sense, right angle regularity for polygonized buildings, and final mesh complexity. Experiments show that adding the right angle objective yields better representations quantitatively and qualitatively than previous work and commonly used polygon generalization methods in remote sensing literature for similar number of vertices.

Fusion of aerial optical and LiDAR data has been a popular problem in remote sensing as they carry complementary information for object detection. We describe a stratified method that involves separately thresholding the normalized digital surface model derived from LiDAR data and the normalized difference vegetation index derived from spectral bands to obtain candidate image parts that contain different object classes, and incorporates spectral and height data with spatial information in a graph cut framework to segment the rest of the image where such separation is not possible. Experiments using a benchmark data set show that the performance of the proposed method that uses small amount of supervision is compatible with the ones in the literature.

Multi-instance multi-label (MIML) learning is a challenging problem in many aspects. Such learning approaches might be useful for many medical diagnosis applications including breast cancer detection and classification. In this study subset of digiPATH dataset (whole slide digital breast cancer histopathology images) are used for training and evaluation of six state-of-the-art MIML methods. At the end, performance comparison of these approaches are given by means of effective evaluation metrics. It is shown that MIML-kNN achieve the best performance that is %65.3 average precision, where most of other methods attain acceptable results as well.