Applied research presented in this paper describes an approach to provide meaningful evaluation of the Machine Learning (ML) components in a Full Motion Video (FMV) Machine Learning Enabled System ( MLES). The MLES itself is not discussed in the paper. We focus on the experimental activity that has been designed to provide confidence that the MLES, when fielded under dynamic and uncertain conditions, performance will not be undermined by a lack of ML robustness. For example, to real-world changes of the same scene under differing light conditions. The paper details the technical approach and how it is applied to data, across the overall experimental pipeline, consisting of a perturbation engine, test pipeline and metric production. Data is from a small imagery dataset and the results are shown and discussed as part of a proof of concept study.

A broad diversity of biological organisms and systems interact with soil in ways that facilitate their growth and survival. These interactions are made possible by strategies that enable organisms to accomplish functions that can be analogous to those required in geotechnical engineering systems. Examples include anchorage in soft and weak ground, penetration into hard and stiff subsurface materials and movement in loose sand. Since the biological strategies have been ‘vetted’ by the process of natural selection, and the functions they accomplish are governed by the same physical laws in both the natural and engineered environments, they represent a unique source of principles and design ideas for addressing geotechnical challenges. Prior to implementation as engineering solutions, however, the differences in spatial and temporal scales and material properties between the biological environment and engineered system must be addressed. Current bio-inspired geotechnics research is addressing topics such as soil excavation and penetration, soil–structure interface shearing, load transfer between foundation and anchorage elements and soils, and mass and thermal transport, having gained inspiration from organisms such as worms, clams, ants, termites, fish, snakes and plant roots. This work highlights the potential benefits to both geotechnical engineering through new or improved solutions and biology through understanding of mechanisms as a result of cross-disciplinary interactions and collaborations.

Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this article aims to develop a coverage-guided testing approach to systematically exploit the internal behavior of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short-term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both stepwise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool testRNN, which is then evaluated on a set of LSTM benchmarks. Experiments confirm that testRNN has advantages over the state-of-the-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, testRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step toward interpretable neural network testing.

River bank slopes incised into Champlain Sea Clay are particularly susceptible to highly retrogressive landslides such as flows and lateral spreads. The concept of progressive failure indicates that a growing failure surface requires some finite deformation prior to instability. However, given the brittleness exhibited by sensitive clays, it is currently unclear whether the pre-failure displacements are sufficiently large to exceed the detection limit of current surface displacement techniques prior to failure. A 5-year monitoring program using UAV-based photogrammetric and conventional total station surveying of a network of ground control markers on an 11 m high slope inclined at 3H:1 V was conducted to assess whether surface displacement measurements can serve as an indicator of potential expansion of a progressive failure surface. After initiation of the monitoring program in 2009, surface displacements indicated a slow accumulation of deformation towards the crest scarp leading up to retrogression events in the form of flow failures in 2012 and 2013. This data indicates that, for the case of episodically retrogressing landslides in the Champlain Sea Clay within the urban boundary of the City of Ottawa, sufficient pre-failure deformations exist to be measurable precursors to progressive failure.

A multi-year cone penetration testing program was conducted at a landslide subject to episodic retrogression in Mud Creek, Ottawa, to assess whether a hand-operated mobile cone penetration test (CPT) could yield new insights into the current degree of remolding under progressive failure in metastable areas of a landslide where conventional tracked rigs are unable to gain access. The mobile CPT rig permitted tests to be performed through the entire thickness of the Champlain Sea deposit at a penetration rate of 0.5 cm/s, with similar results to tests performed at the standard 2 cm/s. Measurements of pore pressure varied considerably with cone size, with the magnitude of pore pressure response decreasing with cone size. The elevation of the slip surface was identified in the tip resistance as the point of transition between the remolded soil above the slip surface and the intact soil below the slip surface, whereas a further 0.5 m of penetration was required to elevate pore pressures to values indicative of the intact soil behaviour. In situ measurements of shear strength of corresponding layers between the intact and remolded profiles to be compared indicated that the soil above the slip surface had remolded to 50% of its fully remolded strength.

An ever-growing range of technologies can be used to implement programmable content. Examples include highly complex System-On-Chip designs, which use traditional electronics, as well as approaches based on quantum technologies and biological systems. Standards and guidance are required to support the safe use of these, and other, emerging programmable technologies. However, technology-specific guidance is challenging to produce, especially in a timely manner. To help bridge this gap, we propose a set of generic assurance topics, which are applicable to all types of programmable content, introducing considerations based on the assurance of both a program, and the associated substrate. The topics are initially introduced by considering multi-core processors. Their application to alternate technologies is illustrated by considering electronic hardware tailored for machine learning, quantum computing and computation using a bio-based substrate.

Intensive research has been conducted on the verification and validation of deep neural networks (DNNs), aiming to understand if, and how, DNNs can be applied to safety critical applications. However, existing verification and validation techniques are limited by their scalability, over both the size of the DNN and the size of the dataset. In this paper, we propose a novel abstraction method which abstracts a DNN and a dataset into a Bayesian network (BN). We make use of dimensionality reduction techniques to identify hidden features that have been learned by hidden layers of the DNN, and associate each hidden feature with a node of the BN. On this BN, we can conduct probabilistic inference to understand the behaviours of the DNN processing data. More importantly, we can derive a runtime monitoring approach to detect in operational time rare inputs and covariate shift of the input data. We can also adapt existing structural coverage-guided testing techniques (i.e., based on low-level elements of the DNN such as neurons), in order to generate test cases that better exercise hidden features. We implement and evaluate the BN abstraction technique using our DeepConcolic tool available at https://github.com/TrustAI/DeepConcolic.

This paper studies the reliability of a real-world learning-enabled system, which conducts dynamic vehicle tracking based on a high-resolution wide-area motion imagery input. The system consists of multiple neural network components to process the imagery inputs - and multiple symbolic (Kalman filter) components - to analyse the processed information for vehicle tracking. It is known that neural networks suffer from adversarial examples, which make them lack robustness. However, it is unclear if and how the adversarial examples over learning components can affect the overall system-level reliability. By integrating a coverage-guided neural network testing tool, DeepConcolic, with the vehicle tracking system, we found that (1) the overall system can be resilient to some adversarial examples thanks to the existence of other components, and (2) the overall system presents an extra level of uncertainty which cannot be determined by analysing the deep learning components only. This research suggests the need for novel verification and validation methods for learning-enabled systems.

There are many aspects to the safe use of artificial intelligence. To date, comparatively little attention has been given to the specialist computational hardware that is used, especially within embedded systems. Consequently, there is a need to identify evidence that would support a compelling assurance argument for the safe use of off-the-shelf, large scale, complex system-on-chip designs. To that end, we summarise issues related to the use of multi-core processors in aviation, which contextualises our problem. We also discuss a collection of considerations that provide evidence to support a compelling assurance argument.

Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative – it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.

In the past few years, significant progress has been made on deep neural networks (DNNs) in achieving human-level performance on several long-standing tasks. With the broader deployment of DNNs on various applications, the concerns over their safety and trustworthiness have been raised in public, especially after the widely reported fatal incidents involving self-driving cars. Research to address these concerns is particularly active, with a significant number of papers released in the past few years. This survey paper conducts a review of the current research effort into making DNNs safe and trustworthy, by focusing on four aspects: verification, testing, adversarial attack and defence, and interpretability. In total, we survey 202 papers, most of which were published after 2017.

We study for the first time the verification problem on learning-enabled state estimation systems for robotics, which use Bayes filter for localisation, and use deep neural network to process sensory input into observations for the Bayes filter. Specifically, we are interested in a robustness property of the systems: given a certain ability to an adversary for it to attack the neural network without being noticed, whether or not the state estimation system is able to function with only minor loss of localisation precision? For verification purposes, we reduce the state estimation systems to a novel class of labelled transition systems with payoffs and partial order relations, and formally express the robustness property as a constrained optimisation objective. Based on this, practical verification algorithms are developed. As a major case study, we work with a real-world dynamic tracking system that uses a Kalman filter (a special case of the Bayes filter) to localise and track a ground vehicle. Its perception system, based on convolutional neural networks, processes a high-resolution Wide Area Motion Imagery (WAMI) data stream. Experimental results show that our algorithms can not only verify the robustness of the WAMI tracking system but also provide useful counterexamples.

Deep neural networks (DNNs) have been deployed in a wide range of applications. We introduce a DNN testing and debugging tool, called DeepConcolic, which is able to detect errors with sufficient rigour so as to be applicable to the testing of DNNs in safety-related applications. DeepConcolic is the first tool that implements a concolic testing technique for DNNs, and the first testing tool that provides users with the functionality of investigating particular parts of a DNN. The tool has been made publicly available and a demo video can be found at https://youtu.be/rliynbhoNLM.

Deep Neural Networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we propose four novel test criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that the generated test inputs, guided by our coverage criteria, are able to capture undesirable behaviours in DNNs. Test cases are generated using both a symbolic approach and a gradient-based heuristic. Our experiments are conducted on state-of-the-art DNNs, obtained using the MNIST and ImageNet datasets.

Recurrent neural networks (RNNs) have been widely applied to various sequential tasks such as text processing, video recognition, and molecular property prediction. We introduce the first coverage-guided testing tool, coined testRNN, for the verification and validation of a major class of RNNs, long short-term memory networks (LSTMs). The tool implements a generic mutation-based test case generation method, and it empirically evaluates the robustness of a network using three novel LSTM structural test coverage metrics. Moreover, it is able to help the model designer go through the internal data flow processing of the LSTM layer. The tool is available through: https://github.com/TrustAI/testRNN under the BSD 3-Clause licence.

In the past few years, significant progress has been made on deep neural networks (DNNs) in achieving human-level intelligence on several long-standing tasks. With broader deployment of DNNs on various applications, the concerns on its safety and trustworthiness have been raised, particularly after the fatal incidents of self-driving cars. Research to address these concerns is very active, with many papers released in the past few years. This survey paper is to conduct a review of the current research efforts on making DNNs safe and trustworthy, by focusing on four aspects, i.e., verification, testing, adversarial attack and defence, and interpretability. In total, we surveyed 178 papers, most of which were published in the most recent two years, i.e., 2017 and 2018.

Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we propose a family of four novel test criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that the generated test inputs guided via our proposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods, we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples) and the computational cost of test case generation. Our experiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and ImageNet.