Coupling (semi-)autonomous drones with on-ground personnel can help improve metrics like mission safety, task effectiveness, and task completion time. However, in order for a drone to be an effective companion, it needs to be able to make intelligent decisions about what to do in a partially observable and dynamic environment in light of uncertainty and multiple competing criteria. One simple example is where and how to move. These kinds of continuous or waypoint-based decisions vary greatly from task to task, such as in the scenario of building a 3D map of an area, getting a minimum number of pixels on objects for automatic target detection, exploring an area around a search team, etc. While it is possible to implement each behavior from scratch, we discuss a flexible and extensible framework that allows the specification of dynamic, controlled, and explainable behaviors based on the multi-criteria decision making (MCDM), an aggregation task, of different UFOMap voxel map layers. While we currently employ specific layers such as drone position, time since a voxel was last observed, minimum distance to a voxel, and exploration fringe, future additional layers present the opportunity for the creation of more complex and novel behaviors. Through testing with simulated flights, we have demonstrated that such an approach is feasible for the construction of useful semi-autonomous behaviors in the pursuit of human-robot teaming.

In the physical universe, truth for computer vision (CV) is impractical if not impossible to obtain. As a result, the CV community has resorted to qualitative practices and sub-optimal quantitative measures. This is problematic because it limits our ability to train, evaluate, and ultimately understand algorithms such as single image depth estimation (SIDE) and structure from motion (SfM). How good are these algorithms, individually and relatively, and where do they break? Herein, we discuss that while truth evades both the real and simulated (SIM) universes, a SIM CV gold-standard can be achieved. We outline an extensible SIM framework and data collection workflow using Unreal Engine with the Robot Operating System (ROS) for three dimensional mapping on low altitude aerial vehicles. Furthermore, voxel-based mapping measures from algorithm output to a SIM gold-standard are discussed. The proposed metrics are demonstrated by analyzing performance across changes in platform context. Ultimately, the current article is a step towards an improved process for comparing algorithms, evaluating their strengths and weaknesses, and automating algorithm design.

Monocular depth estimation methods using structure from motion (SfM) have become increasingly capable of reconstructing 3D representations from a sequence of 2D images. In the context of unmanned aerial vehicles (UAVs), these techniques can be used to create an occupancy map of an environment, which is useful for planning and navigation. OctoMap and a recent improvement, UFOMap, are commonly used hierarchical representations that represent the value of a voxel cell as the probability of being occupied. Although this captures some uncertainty in the map and allows for dynamic updates, it does not fully utilize the known characteristics of the sensor and SfM algorithm, and it can lead to unnecessarily noisy results. In this paper, we propose an approach to assign a weight to each point in a point cloud update based on camera extrinsics and SfM confidence. The weighted points are then added to the voxel map in a way that more closely resembles a degree of confidence rather than a probability. In this way, we take the first steps toward designing a fuzzy voxel map that is more robust in noisy situations and captures useful uncertainty to help with UAV applications. We demonstrate our approach on simulated scenarios using Unreal Engine and AirSim.

There is a severe demand for, and shortage of, large accurately labeled datasets to train supervised computational intelligence (CI) algorithms in domains like unmanned aerial systems (UAS) and autonomous vehicles. This has hindered our ability to develop and deploy various computer vision algorithms in/across environments and niche domains for tasks like detection, localization, and tracking. Herein, we propose a new human-in-the-loop (HITL) based growing neural gas (GNG) algorithm to minimize human intervention during labeling large UAS data collections over a shared geospatial area. Specifically, we address human driven events like new class identification and mistake correction. We also address algorithm-centric operations like new pattern discovery and self-supervised labeling. The effectiveness of our algorithm is demonstrated using simulated realistic ray-traced low altitude UAS data from the Unreal Engine. Our results show that it is possible to increase speed and reduce mental fatigue over hand labeling large image datasets.

Because of weight, power, and cost constraints, most unmanned aircraft systems (UAS) contain monocular camera systems. Real- time structure from motion (SfM) algorithms are required for monocular UAS systems to sense and autonomously navigate 3D environments. The SfM algorithm must be designed to work near real-time and handle the wide variety of possible extrinsic parameters produced by UAS image pairs. Common rigid epipolar rectification techniques (homography-based rectification) do not accurately model epipolar geometries with epipoles close to or contained within the image bounds. Common UAS movement types, translation along lens axis, tight radial turns, circular patterns with GPS locked camera focus, can all produce epipolar geometries with epipoles inside the camera frame. Using a generalized epipolar rectification technique, all extrinsic UAS movement types can be handled, and optimized image block matching techniques can be used to produce disparity/depth maps. The majority of UASs contain GPS/IMU/magnetometer modules. These modules provide absolute camera extrinsic parameters for every image. The essential and fundamental matrices between image pairs can be calculated from these absolute extrinsics. SfM is performed, depth images are produced, and the camera space pixel values can be projected as point clouds in three-dimensional space. These point clouds provide scene understanding that can be used for autonomous reasoning systems.

The spatial configuration of objects in a scene is important to many applications. In particular, 3D environments constructed from point cloud observations are often used for navigation and planning with real-time requirements. In these settings, the ability to recognize and distinguish one set of objects from another may depend largely on how they are positioned with respect to each other. In this article, we explore two different approaches for describing the relative spatial relationship between two objects represented as 3D points: the histogram of forces, and a method using bounding boxes and fuzzy numbers. We use 2D axis-aligned projections of the objects to facilitate the computation of force histograms, and compare this approach to the bounding box and fuzzy number method. Our experiments are performed on the NPM3D dataset, consisting of hand-labeled point cloud objects in an outdoor street-level environment. The results highlight the strengths and weaknesses of each approach and we discuss the most appropriate applications for both.

A number of real-time object detection, tracking, and autonomy artificial intelligence (AI) and machine learning (ML) algorithms are being proposed for unmanned aerial vehicles (UAVs). A big challenge is can we stress test these algorithms, identify their strengths and weaknesses, and assess if the UAV is safe and trustworthy? The process of collecting real-world UAV data is costly, time consuming, and riddled by lack of quality geospatial ground truth and metadata. Herein, we outline a fully automated framework and work ow to address the above challenges using free or low-cost assets, the photorealistic Unreal Engine (UE), and AirSim aerial platform simulator. Specifically, we discuss the rapid prototyping of an outdoor environment combined with the robotic operating system (ROS) for abstracting UAV data collection, control, and processing. Real and accurate ground truth is collected and metrics are presented for individual frame and entire flight collection evaluation. Metrics recorded and analyzed include percentage of scene mapped, 3D mapping accuracy, time to complete task, object detection and tracking statistics, battery usage, altitude (from ground), collisions, and other statistics. These metrics are computed in general and with respect to context, e.g., clutter, view angle, etc. Overall, the proposed work is an automated way to explore UAV operation before real-world testing or deployment. Promising preliminary results are discussed for an outdoor environment with vegetation, short and long range objects, buildings, people, vehicles, and other features for a UAV performing loitering and interrogation.

In general, there is a severe demand for, and shortage of, large accurately labeled datasets to train supervised machine learning (ML) algorithms for domains like smart cars and unmanned aerial systems (UAS). This impacts a number of real-world problems from standing up ML on niche domains to ML performance in/across different environments. Herein, we consider the task of efficiently, meaning requiring the least amount of human intervention possible, converting large UAS data collections over a shared geospatial area into accurately labeled training data. Herein, we take a human-in-the-loop (HITL) approach that is based on coupling active learning and self-supervised learning to efficiently label low altitude UAS imagery for the goal of training ML algorithms for underlying tasks like detection, localization, and tracking. Specifically, we propose an extension to our stream classification algorithm StreamSoNG based on human intervention. We also extend StreamSoNG to rely on a second and initially more mature, but assumed incomplete, ML classifier. Herein, we use the Unreal Engine to simulate realistic ray-traced low altitude UAS data and facilitate algorithmic performance analysis in a controlled fashion. While our results are preliminary, they suggest that this approach is a good trade off between not overloading a human operator and circumventing fundamental stream classification algorithm limitations.

Low altitude Unmanned Aerial Systems (UASs) provide a highly flexible and capable platform for remote sensing and autonomous control. There are many applications that would benefit from an additional bird’s eye view, including mapping, environmental reconnaissance, and search and rescue to name a few. An autonomous (or partially autonomous) drone could assist in several of these scenarios, freeing the operator to focus on higherlevel strategic planning. While numerous commercial drones exist on the market, none truly provide a flexible foundation for vision guided autonomy research. Herein, we propose the design of a physical UAS platform, called VADER (Visually Aware Drone for Environmental Reconnaissance), and an accompanying simulation environment that addresses many of these tasks. In particular, we show how Commercial Off The Shelf (COTS) hardware and open source software can now be combined to realize powerful end-to-end UAS research solutions. The beauty of unifying these factors is accelerated prototyping and minimal time to migrate and test in the real world. This article outlines VADER and case studies are presented to demonstrate capabilities.

The identification followed by avoidance or removal of explosive hazards in past and/or present conflict zones is a serious threat for both civilian and military personnel. This is a challenging task as variability exists with respect to the objects, their environment and emplacement context, to name a few factors. A goal is the development of automatic or human-in-the-loop sensor technologies that leverage signal processing, data fusion and machine learning. Herein, we explore the detection of side attack explosive hazards (SAEHs) in three dimensional voxel space radar via different shallow and deep convolutional neural network (CNN) architectures. Dimensionality reduction is performed by using multiple projected images versus the raw three dimensional voxel data, which leads to noteworthy savings in input size and associated network hyperparameters. Last, we explore the accuracy and interpretation of solutions learned via random versus intelligent network weight initialization. Experiments are provided on a U.S. Army data set collected over different times, weather conditions, target types and concealments. Preliminary results indicate that deep learning can perform as good as, if not better, than a skilled domain expert, even in light of limited training data with a class imbalance.

Explosive hazards in current and former conflict zones are a serious threat to both civilians and soldiers alike. Significant effort has been dedicated to identifying sensors, algorithms and fusion strategies to detect such threats. However, a challenging aspect of the field is that we are not necessarily at war with the threats (objects). Instead, we are at conflict with people who are constantly evolving their strategies of attack along with their preferred threat. One such method of threat delivery is side attack explosive ballistics (SAEB). In this article, we explore different 3D voxel-space radar signal processing methods for SAEB detection on a U.S. Army provided vehicle-mounted platform. In particular, we explore the fusion of a matched filter (MF) and size contrast filter (SCF). Clustering is applied to the fused result and heuristics are used to reduce the systems false alarm rate. Performance is assessed in the context of receiver operating characteristic (ROC) curves on data from a U.S. Army test site containing multiple target and clutter types, levels of concealment and times of day.

This paper explores some of the issues in working across time and space boundaries for professional development of online educators. It focuses on voluntary aspects of professional development within the context of networked learning communities. It also places an emphasis on ways of respecting and acknowledging cultural and individual difference within such voluntary communities in ways that also respond to the wider needs of others in the community. A sampling of current practice in networked learning communities is offered as a backdrop to a more detailed review of our own collaborative practice and the development of the World Association for Online Education (WAOE) as one example of a virtual community. As a way of describing our design and response to cultural diversity that exists within our community each of us also offers our own individual ‘story’ of our involvement in the collaborative development process. This is offered to illustrate the way that our provision will function and also acknowledge ourselves as a sub-community seeking to research and understand our own collaborative practice.’

Substantial interest resides in identifying sensors, algorithms and fusion theories to detect explosive hazards. This is a significant research effort because it impacts the safety and lives of civilians and soldiers alike. However, a challenging aspect of this field is we are not in conflict with the threats (objects) per se. Instead, we are dealing with people and their changing strategies and preferred method of delivery. Herein, we investigate one method of threat delivery, side attack explosive ballistics (SAEB). In particular, we explore a vehicle-mounted synthetic aperture acoustic (SAA) platform. First, a wide band SAA signal is decomposed into a higher spectral resolution signal. Next, different multi/hyperspectral signal processing techniques are explored for manual band analysis and selection. Last, a convolutional neural network (CNN) is used for filter learning and classification relative to the full signal versus different subbands. Performance is assessed in the context of receiver operating characteristic (ROC) curves on data from a U.S. Army test site that contains multiple target and clutter types, levels of concealment and times of day. Preliminary results indicate that a machine learned CNN solution can achieve better performance than our previously established human engineered Fourier-based Fraz feature with kernel support vector machine classification.

A serious threat to civilians and soldiers is buried and above ground explosive hazards. The automatic detection of such threats is highly desired. Many methods exist for explosive hazard detection, e.g., hand-held based sensors, downward and forward looking vehicle mounted platforms, etc. In addition, multiple sensors are used to tackle this extreme problem, such as radar and infrared (IR) imagery. In this article, we explore the utility of feature and decision level fusion of learned features for forward looking explosive hazard detection in IR imagery. Specifically, we investigate different ways to fuse learned iECO features pre and post multiple kernel (MK) support vector machine (SVM) based classification. Three MK strategies are explored; fixed rule, heuristics and optimization-based. Performance is assessed in the context of receiver operating characteristic (ROC) curves on data from a U.S. Army test site that contains multiple target and clutter types, burial depths and times of day. Specifically, the results reveal two interesting things. First, the different MK strategies appear to indicate that the different iECO individuals are all more-or-less important and there is not a dominant feature. This is reinforcing as our hypothesis was that iECO provides different ways to approach target detection. Last, we observe that while optimization-based MK is mathematically appealing, i.e., it connects the learning of the fusion to the underlying classification problem we are trying to solve, it appears to be highly susceptible to over fitting and simpler, e.g., fixed rule and heuristics approaches help us realize more generalizable iECO solutions.

The remarkable popularity of smartphones over the past decade has led to a technological race for dominance in market share. This has resulted in a flood of new processors and sensors that are inexpensive, low power and high performance. These sensors include accelerometers, gyroscope, barometers and most importantly cameras. This sensor suite, coupled with multicore processors, allows a new community of researchers to build small, high performance platforms for low cost. This paper describes a system using off-the-shelf components to perform position tracking as well as environment modeling. The system relies on tracking using stereo vision and inertial navigation to determine movement of the system as well as create a model of the environment sensed by the system.

Forward looking ground penetrating radar (FLGPR) has the benefit of detecting objects at a significant standoff distance. The FLGPR signal is radiated over a large surface area and the radar signal return is often weak. Improving detection, especially for buried in road targets, while maintaining an acceptable false alarm rate remains to be a challenging task. Various kinds of features have been developed over the years to increase the FLGPR detection performance. This paper focuses on investigating the use of as many features as possible for detecting buried targets and uses the sequential feature selection technique to automatically choose the features that contribute most for improving performance. Experimental results using data collected at a government test site are presented.

Explosive hazards in current and former conflict zones are a threat to both military and civilian personnel. As a result, much effort has been dedicated to identifying automated algorithms and systems to detect these threats. However, robust detection is complicated due to factors like the varied composition and anatomy of such hazards. In order to solve this challenge, a number of platforms (vehicle-based, handheld, etc.) and sensors (infrared, ground penetrating radar, acoustics, etc.) are being explored. In this article, we investigate the detection of side attack explosive ballistics via a vehicle-mounted acoustic sensor. In particular, we explore three acoustic features, one in the time domain and two on synthetic aperture acoustic (SAA) beamformed imagery. The idea is to exploit the varying acoustic frequency profile of a target due to its unique geometry and material composition with respect to different viewing angles. The first two features build their angle specific frequency information using a highly constrained subset of the signal data and the last feature builds its frequency profile using all available signal data for a given region of interest (centered on the candidate target location). Performance is assessed in the context of receiver operating characteristic (ROC) curves on cross-validation experiments for data collected at a U.S. Army test site on different days with multiple target types and clutter. Our preliminary results are encouraging and indicate that the top performing feature is the unrolled two dimensional discrete Fourier transform (DFT) of SAA beamformed imagery.

The U.S. Army RDECOM CERDEC Night Vision and Electronic Sensors Directorate (NVESD) is executing a program to assess the performance of a variety of sensor modalities for standoff detection of roadside explosive hazards. The program objective is to identify an optimal sensor or combination of fused sensors to incorporate with autonomous detection algorithms into a system of systems for use in future route clearance operations. This paper provides an overview of the program, including a description of the sensors under consideration, sensor test events, and ongoing data analysis.

This paper proposes a machine learning algorithm for subsurface object detection on multiple-input-multiple-output (MIMO) forward-looking ground-penetrating radar (FLGPR). By detecting hazards using FLGPR, standoff distances of up to tens of meters can be acquired, but this is at the degradation of performance due to high false alarm rates. The proposed system utilizes an anomaly detection prescreener to identify potential object locations. Alarm locations have multiple one-dimensional (ML) spectral features, two-dimensional (2D) spectral features, and log-Gabor statistic features extracted. The ability of these features to reduce the number of false alarms and increase the probability of detection is evaluated for both co-polarizations present in the Akela MIMO array. Classification is performed by a Support Vector Machine (SVM) with lane-based cross-validation for training and testing. Class imbalance and optimized SVM kernel parameters are considered during classifier training.