Collecting and annotating real-world data for the development of object detection models is a time-consuming and expensive process. In the military domain in particular, data collection can also be dangerous or infeasible. Training models on synthetic data may provide a solution for cases where access to real-world training data is restricted. However, bridging the reality gap between synthetic and real data remains a challenge. Existing methods usually build on top of baseline Convolutional Neural Network (CNN) models that have been shown to perform well when trained on real data, but have limited ability to perform well when trained on synthetic data. For example, some architectures allow for fine-tuning with the expectation of large quantities of training data and are prone to overfitting on synthetic data. Related work usually ignores various best practices from object detection on real data, e.g. by training on synthetic data from a single environment with relatively little variation. In this paper we propose a methodology for improving the performance of a pre-trained object detector when training on synthetic data. Our approach focuses on extracting the salient information from synthetic data without forgetting useful features learned from pre-training on real images. Based on the state of the art, we incorporate data augmentation methods and a Transformer backbone. Besides reaching relatively strong performance without any specialized synthetic data transfer methods, we show that our methods improve the state of the art on synthetic data trained object detection for the RarePlanes and DGTA-VisDrone datasets, and reach near-perfect performance on an in-house vehicle detection dataset.

Combining data from multiple sensors to improve the overall robustness and reliability of a classification system has become crucial in many applications, from military surveillance and decision support, to autonomous driving, robotics, and medical imaging. This so-called sensor fusion is especially interesting for fine-grained target classification, in which very specific sub-categories (e.g. ship types) need to be distinguished, a task that can be challenging with data from a single modality. Typical modalities are electro-optical (EO) image sensors, that can provide rich visual details of an object of interest, and radar, that can yield additional spatial information. Defined by the approach used to combine data from these sensors, several fusion techniques exist. For example, late fusion can merge class probabilities outputted by separate processing pipelines dedicated to each of the individual sensor data. In particular, deep learning (DL) has been widely leveraged for EO image analysis, but typically requires a lot of data to adapt to the nuances of a fine-grained classification task. Recent advances in DL on foundation models have shown a high potential when dealing with in-domain data scarcity, especially in combination with few-shot learning. This paper presents a framework to effectively combine EO and radar sensor data, and shows how this method outperforms stand-alone single sensor methods for fine-grained target classification. We adopt a strong few-shot image classification baseline based on foundation models, which robustly handles the lack of in-domain data and exploits rich visual features. In addition, we investigate a weighted and a Bayesian fusion approach to combine target class probabilities outputted by the image classification model and radar kinematic features. Experiments with data acquired in a measurement campaign at the port of Rotterdam show that our fusion method improves on the classification performance of individual modalities.

Deep learning has become the preferred method for automated object detection, but the accurate detection of small objects remains a challenge due to the lack of distinctive appearance features. Most deep learning-based detectors do not exploit the temporal information that is available in video, even though this context is often essential when the signal-to-noise ratio is low. In addition, model development choices, such as the loss function, are typically designed around medium-sized objects. Moreover, most datasets that are acquired for the development of small object detectors are task-specific and lack diversity, and the smallest objects are often not well annotated. In this study, we address the aforementioned challenges and create a deep learning-based pipeline for versatile small object detection. With an in-house dataset consisting of civilian and military objects, we achieve a substantial improvement in YOLOv8 (baseline mAP = 0.465) by leveraging the temporal context in video and data augmentations specifically tailored to small objects (mAP = 0.839). We also show the benefit of having a carefully curated dataset in comparison with public datasets and find that a model trained on a diverse dataset outperforms environment-specific models. Our findings indicate that small objects can be detected accurately in a wide range of environments while leveraging the speed of the YOLO architecture.

Multiple object tracking (MOT) interest has grown in recent years, both in civil and military contexts, enhancing situational awareness for better decision-making. Typically, state-of-the-art methods integrate motion and appearance features to preserve the trajectory of each object over time, using new detection information when available. Visual features are fundamental when it comes to solving temporary occlusion or complex trajectories, i.e. non-linear motion associated with high object speeds or low framerate. Currently, these features are extracted by powerful deep learning-based models trained on the re-identification (ReID) task. However, research focuses mostly on scenarios involving pedestrians or vehicles, limiting the adaptability and transferability of such methods to other use cases. In this paper we investigate the added value of a variety of appearance features for comparing vessel appearance. We also include recent advances in foundation models that show their out-of-the-box applicability to unseen circumstances. Finally, we discuss how the robust visual features could improve multiple object tracking performances in the specialized domain of maritime surveillance.

Semi-supervised techniques have been successful in reducing the amount of labels needed to train a neural network. Often these techniques focus on making the most out of the given labels and exploiting the unlabeled data. Instead of considering the labels as a given, we start with focusing on how to select good labels for efficient semi-supervised learning with pseudo-labels. We propose CLaP: Clustering, Label selection and Pseudo-labels. It clusters an unlabeled dataset on an embedding, that was pretrained on large scale dataset (ImageNet1k). We use the cluster centers for querying initial labels that are both representative and diverse. We use samples consistent over multiple clustering runs as pseudo-labels. We propose a mixed loss on both the initial labels and the pseudo-labeled data to train a neural network. With CLaP the samples to be annotated are automatically selected, reducing the load by a human annotator to search for these samples. We demonstrate that CLaP outperforms of state-of-the-art methods in few-shot transfer tasks on full datasets by 20% on 1-shot to 3% on 5-shot. It also outperforms the accuracy of the state-of-the-art in the BSCD-FSL benchmark up to 23%, depending on the dataset and amount of labels.

ABSTRACT This paper proposes an automated approach to predict crack pattern similarities that correlate well with assessment by structural engineers. We use Siamese convolutional neural networks (SCNN) that take two crack pattern images as inputs and output scalar similarity measures. We focus on 2D masonry facades with and without openings. The image pairs are generated using a statistics-based approach and labelled by 28 structural engineering experts. When the data is randomly split into fit and test data, the SCNNs can achieve good performance on the test data ( ). When the SCNNs are tested on ”unseen” archetypes, their test values are on average 1% lower than the case where all archetypes are ”seen” during the training. These very good results indicate that SCNNs can generalise to unseen cases without compromising their performance. Although the analyses are restricted to the considered synthetic images, the results are promising and the approach is general.

Electro-optical (EO) sensors are essential for surveillance in military and security applications. Recent technological advancements, especially the developments in Deep Learning (DL), have enabled improved object detection and tracking in complex and dynamic environments. Most of this research focuses on readily available visible light (VIS) images. To apply these technologies for Thermal infrared (TIR) imagery, DL networks can be retrained using image data in the TIR domain. However, such a training set with enough samples is not easily available. This paper presents an unsupervised domain adaptation method for ship detection in TIR imagery using paired VIS and TIR images. The proposed method leverages on the pairing of VIS and TIR images and performs domain adaptation using detections in the VIS imagery as ground-truth to provide data for the TIR domain learning. The method performs ship detection from the VIS images using a pretrained convolutional neural network (CNN). These detections are subsequently improved using a tracking algorithm. The proposed TIR object detection model follows a two- stage training process. In the first stage, the model's head is trained, which consists of the regression layers that output the bounding boxes of the detected objects. In the second stage, the model's feature extractor is trained to learn more discriminative features. The method is evaluated on a dataset of recordings at Rotterdam harbor. Experiments demonstrate that the resulting TIR detector performs comparably with its VIS counterpart, in addition to providing reliable detections in adverse environmental conditions where VIS model fails. The proposed method has significant potential for real-world applications, including maritime surveillance.

Automated object detection is becoming more relevant in a wide variety of applications in the military domain. This includes the detection of drones, ships, and vehicles in video and IR video. In recent years, deep learning based object detection methods, such as YOLO, have shown to be promising in many applications for object detection. However, current methods have limited success when objects of interest are small in number of pixels, e.g. objects far away or small objects closer by. This is important, since accurate small object detection translates to early detection and the earlier an object is detected the more time is available for action. In this study, we investigate novel image analysis techniques that are designed to address some of the challenges of (very) small object detection by taking into account temporal information. We implement six methods, of which three are based on deep learning and use the temporal context of a set of frames within a video. The methods consider neighboring frames when detecting objects, either by stacking them as additional channels or by considering difference maps. We compare these spatio-temporal deep learning methods with YOLO-v8 that only considers single frames and two traditional moving object detection methods. Evaluation is done on a set of videos that encompasses a wide variety of challenges, including various objects, scenes, and acquisition conditions to show real-world performance.

antic segmentation of aerial images is a critical task in various domains such as urban planning, and monitoring deforestation or critical infrastructure. However, the annotation process required for training accurate segmentation models is often time-consuming and labor-intensive. This paper presents a novel approach to address this challenge by leveraging the power of clustering techniques applied to the embeddings obtained from a SimCLRv2 model pretrained on the ImageNet dataset. By using this clustering approach, fewer training samples are needed, and the annotation only needs to be done for each cluster instead of each pixel in the image, significantly reducing the annotation time. Our proposed method uses SimCLRv2 to obtain rich feature representations (embeddings) from a dataset of unlabeled aerial images. These embeddings are then subjected to clustering, enabling the grouping of semantically similar image regions. In addition to directly using these clusters as class labels, we can treat these clusters as pseudo-classes, allowing us to construct a pseudo-label dataset for fine-tuning a segmentation network. Through experiments conducted on two benchmark aerial image datasets (Potsdam and Vaihingen), we demonstrate the effectiveness of our approach in achieving segmentation results in line with similar works on few-shot segmentation while significantly reducing the annotation effort required, thereby highlighting its practical applicability. Overall, the combination of SimCLRv2 embeddings and clustering techniques presents a promising avenue for achieving accurate image segmentation while minimizing the annotation burden, making it highly relevant for remote sensing applications and aerial imagery analysis.

Visual Question Answering (VQA) is a very interesting technique to answer natural language questions about an image. Recent methods have focused on incorporating knowledge into an improved VQA model, by augmenting the training set, representing scene graphs, or including reasoning. We also leverage knowledge to make VQA more robust. Yet we take a different route: we take the VQA model as-is and extend it with a novel algorithm called Guided-VQA that guides the questioning by leveraging knowledge to obtain better answers. This enables knowledge-extended VQA while not having to retrain the VQA model. This is beneficial when computing resources and/or time to adapt to new knowledge are limited. We start with the observation that VQA has difficulties with answering compositional and finegrained questions. We propose to solve this by a coarse-to-fine scheme of posing questions. The proposed Guided-VQA algorithm is an iterative, conditional refinement that decomposes a compositional, finegrained question into a sequence of coarse-to-fine questions by leveraging taxonomic knowledge about the involved objects. On Visual Genome, we show that it improves the answers significantly over standard VQA. This is relevant for robust deployment of VQA where resources or adaptation time are limited.

Learning object detection models with a few labels, is possible due to ingenious few-shot techniques, and due to clever selection of images to be labeled. Few-shot techniques work with as few as 1 to 10 randomized labels per object class. We are curious if performance of randomized label selection can be improved by selecting 1 to 10 labels per object class in a non-random manner. Several active learning techniques have been proposed to select object labels, but all started with a minimum of several tens of labels. We explore an effective and simple label selection strategy, for the case of 1 to 10 labels per object class. First, the full unlabeled dataset is clustered into N clusters, where N is the desired number of labels. Clustering is based on k-means on embedding vectors from a state-of-the-art pretrained image classification model (SimCLR v2). The image closest to the center is selected to be labeled. It is effective: on Pascal VOC we validate that it improves over randomized selection over 25%, with large improvements especially when having 1 label per object class. We have several benefits to report on this simple strategy: it is easy to implement, it is effective, and it is relevant in practice where one often starts with a dataset without any labels.

For the segmentation of magnetic resonance brain images into anatomical regions, numerous fully automated methods have been proposed and compared to reference segmentations obtained manually. However, systematic differences might exist between the resulting segmentations, depending on the segmentation method and underlying brain atlas. This potentially results in sensitivity differences to disease and can further complicate the comparison of individual patients to normative data. In this study, we aim to answer two research questions: 1) to what extent are methods interchangeable, as long as the same method is being used for computing normative volume distributions and patient-specific volumes? and 2) can different methods be used for computing normative volume distributions and assessing patient-specific volumes? To answer these questions, we compared volumes of six brain regions calculated by five state-of-the-art segmentation methods: Erasmus MC (EMC), FreeSurfer (FS), geodesic information flows (GIF), multi-atlas label propagation with expectation–maximization (MALP-EM), and model-based brain segmentation (MBS). We applied the methods on 988 non-demented (ND) subjects and computed the correlation (PCC-v) and absolute agreement (ICC-v) on the volumes. For most regions, the PCC-v was good ( > 0.75 ), indicating that volume differences between methods in ND subjects are mainly due to systematic differences. The ICC-v was generally lower, especially for the smaller regions, indicating that it is essential that the same method is used to generate normative and patient data. To evaluate the impact on single-subject analysis, we also applied the methods to 42 patients with Alzheimer’s disease (AD). In the case where the normative distributions and the patient-specific volumes were calculated by the same method, the patient’s distance to the normative distribution was assessed with the z-score. We determined the diagnostic value of this z-score, which showed to be consistent across methods. The absolute agreement on the AD patients’ z-scores was high for regions of thalamus and putamen. This is encouraging as it indicates that the studied methods are interchangeable for these regions. For regions such as the hippocampus, amygdala, caudate nucleus and accumbens, and globus pallidus, not all method combinations showed a high ICC-z. Whether two methods are indeed interchangeable should be confirmed for the specific application and dataset of interest.

The degree of similarity between damage patterns often correlates with the likelihood of having similar damage causes. Therefore, deciding whether crack patterns are similar is one of the key steps in assessing the conditions of masonry structures. To our knowledge, no literature has been published regarding masonry crack pattern similarity measures that would correlate well with assessment by structural engineers. Hence, currently, similarity assessments are solely performed by experts and require considerable time and effort. Moreover, it is expensive, limited by the availability of experts, and yields only qualitative answers. In this work, we propose an automated approach that has the potential to overcome the above shortcomings and perform comparably with experts. At its core is a deep neural network embedding that can be used to calculate a numerical distance between crack patterns on comparable facades. The embedding is obtained from fitting a deep neural network to perform a classification task; i.e., to predict the crack pattern archetype label from a crack pattern image. The network is fitted to synthetic crack patterns simulated using a statistics-based approach proposed in this work. The simulation process can account for important crack pattern characteristics such as crack location, orientation, and length. The embedding transforms a crack pattern (raster image) into a 64-dimensional real-valued vector space where the closeness between two vectors is calculated as the cosine of their angle. The proposed approach is tested on 2D facades with and without openings, and with synthetic crack patterns that consist of a single crack and multiple cracks.

Early threat assessment of vessels is an important surveillance task during naval operations. Whether a vessel is a threat depends on a number of aspects. Amongst those are the vessel class, the closest point of approach (CPA), the speed and direction of the vessel and the presence of possible threatening items on board the vessel such as weapons. Currently, most of these aspects are observed by operators viewing the camera imagery. Whether a vessel is a potential threat will depend on the final assessment of the operator. Automated analysis of electro-optical (EO) imagery for aspects of potential threats during surveillance can support the operator during observation. This can release the operator from continuous guard and provide him with the tools to provide a better overview of possible threats in the surroundings during a surveillance task. In this work, we apply different processing algorithms, including detection, tracking and classification, on recorded multi-band EO imagery in a harbor environment with many small vessels. With the results we aim to automatically determine the vessel’s CPA, number of people on board and the presence of possibly threatening items on board of the vessel. Hereby we show that our algorithms can support the operator in assessing whether a vessel poses a threat or not.

Automatic detection and tracking of persons and vehicles can greatly increase situational awareness in many military applications. Various methods for detection and tracking have been proposed so far, both for rule-based and learning approaches. With the advent of deep learning, learning approaches generally outperform rule-based approaches. Pre-trained neural networks on datasets like MS COCO can give reasonable detection performance on military datasets. However, for optimal performance it is advised to optimize the training of these pre-trained networks with a representative dataset. In typical military settings, it is a challenge to acquire enough data, and to split the training and test set properly. In this paper we evaluate fine-tuning on military data and compare different pre- and post-processing methods. First we compare a standard pre-trained RetinaNet detector with a fine-tuned version, trained on similar objects, which are recorded at distances different than the distance in the test set. On the aspect of distance this train set is therefore out-of-distribution. Next, we augment the training examples by both increasing and decreasing their size. Once detected, we use a template tracker to follow the objects, compensating for any missing detections. We show the results on detection and tracking of persons and vehicles in visible imagery in a military long range detection setting. The results show the added value of fine-tuning a neural net with augmented examples, where final network performance is similar to human visual performance for detection of targets, with a target area of tens of pixels in a moderately cluttered land environment.

BackgroundIdentifying persons at risk for cognitive decline may aid in early detection of persons at risk of dementia and to select those that would benefit most from therapeutic or preventive measures for dementia.ObjectiveIn this study we aimed to validate whether cognitive decline in the general population can be predicted with multivariate data using a previously proposed supervised classification method: Disease State Index (DSI).MethodsWe included 2,542 participants, non-demented and without mild cognitive impairment at baseline, from the population-based Rotterdam Study (mean age 60.9 ± 9.1 years). Participants with significant global cognitive decline were defined as the 5% of participants with the largest cognitive decline per year. We trained DSI to predict occurrence of significant global cognitive decline using a large variety of baseline features, including magnetic resonance imaging (MRI) features, cardiovascular risk factors, APOE-ε4 allele carriership, gait features, education, and baseline cognitive function as predictors. The prediction performance was assessed as area under the receiver operating characteristic curve (AUC), using 500 repetitions of 2-fold cross-validation experiments, in which (a randomly selected) half of the data was used for training and the other half for testing.ResultsA mean AUC (95% confidence interval) for DSI prediction was 0.78 (0.77–0.79) using only age as input feature. When using all available features, a mean AUC of 0.77 (0.75–0.78) was obtained. Without age, and with age-corrected features and feature selection on MRI features, a mean AUC of 0.70 (0.63–0.76) was obtained, showing the potential of other features besides age.ConclusionThe best performance in the prediction of global cognitive decline in the general population by DSI was obtained using only age as input feature. Other features showed potential, but did not improve prediction. Future studies should evaluate whether the performance could be improved by new features, e.g., longitudinal features, and other prediction methods.

Imaging systems can be used to obtain situational awareness in maritime situations. Important tools for these systems are automatic detection and tracking of objects in the acquired imagery, in which numerous methods are being developed. When designing a detection or tracking algorithm, its quality should be ensured by a comparison with existing algorithms and/or with a ground truth. Detection and tracking methods are often designed for a specific task, so evaluation with respect to this task is crucial, which demands for different evaluation measures for different tasks. We, therefore, propose a variety of quantitative measures for the performance evaluation of detectors and trackers for a variety of tasks. The proposed measures are a rich set from which an algorithm designer can choose in order to optimally design and assess a detection or tracking algorithm for a specific task. We compare these different evaluation measures by using them to assess detection and tracking quality in different maritime detection and tracking situations, obtained from three real-life infrared video data sets. With the proposed set of evaluation measures, a user is able to quantitatively assess the performance of a detector or tracker, which enables an optimal design for his approach. (C) 2019 Society of Photo-Optical Instrumentation Engineers (SPIE)

Brain imaging data are increasingly made publicly accessible, and volumetric imaging measures derived from population-based cohorts may serve as normative data for individual patient diagnostic assessment. Yet, these normative cohorts are usually not a perfect reflection of a patient's base population, nor are imaging parameters such as field strength or scanner type similar. In this proof of principle study, we assessed differences between reference curves of subcortical structure volumes of normal controls derived from two population-based studies and a case-control study. We assessed the impact of any differences on individual assessment of brain structure volumes. Percentile curves were fitted on the three healthy cohorts. Next, percentile values for these subcortical structures for individual patients from these three cohorts, 91 mild cognitive impairment and 95 Alzheimer's disease cases and patients from the Alzheimer Center, were calculated, based on the distributions of each of the three cohorts. Overall, we found that the subcortical volume normative data from these cohorts are highly interchangeable, suggesting more flexibility in clinical implementation.

Multichannel image registration is an important challenge in medical image analysis. Multichannel images result from modalities such as dual-energy CT or multispectral microscopy. Besides, multichannel feature images can be derived from acquired images, for instance, by applying multiscale feature banks to the original images to register. Multichannel registration techniques have been proposed, but most of them are applicable to only two multichannel images at a time. In the present study, we propose to formulate multichannel registration as a groupwise image registration problem. In this way, we derive a method that allows the registration of two or more multichannel images in a fully symmetric manner (i.e., all images play the same role in the registration procedure), and therefore, has transitive consistency by definition. The method that we introduce is applicable to any number of multichannel images, any number of channels per image, and it allows to take into account correlation between any pair of images and not just corresponding channels. In addition, it is fully modular in terms of dissimilarity measure, transformation model, regularisation method, and optimisation strategy. For two multimodal datasets, we computed feature images from the initially acquired images, and applied the proposed registration technique to the newly created sets of multichannel images. MIND descriptors were used as feature images, and we chose total correlation as groupwise dissimilarity measure. Results show that groupwise multichannel image registration is a competitive alternative to the pairwise multichannel scheme, in terms of registration accuracy and insensitivity towards registration reference spaces.