Neural Network pruning is commonly used to reduce the size of a neural network, reducing the memory footprint, while maintaining an acceptable loss. However, currently the only approach explored for removing a parameter from a neural network is to remove the parameter suddenly, irrespective of the pruning method, be it one-shot, iterative or sparsity-induced. We hypothesize that this sudden removal will cause the loss of the information contained within the removed parameters, information which could be useful when retraining the neural network after pruning. To resolve this, we propose Soft Pruning, a method of slowly decaying parameters out of a neural network. We compare this to one-shot pruning on the vision-based tasks of classification, autoencoding, and latent space dimensionality reduction. In every experiment, Soft Pruning is able to match or outperform one-shot pruning; in classification, Soft Pruning enables pruning to significantly greater extents than one-shot pruning, retaining over 60% accuracy where one-shot pruning becomes equivalent to random guessing. In autoencoding, Soft Pruning is able to achieve up to 17% lower loss after pruning. Finally, applied to latent space dimensionality reduction, Soft Pruning is shown to achieve more than 60% lower loss compared to one-shot pruning.

This study addresses the problem of hand–eye calibration in robotic systems by developing Continual Learning (CL)-based approaches. Traditionally, robots require explicit models to transfer knowledge from camera observations to their hands or base. However, this poses limitations, as the hand–eye calibration parameters are typically valid only for the current camera configuration. We, therefore, propose a flexible and autonomous hand–eye calibration system that can adapt to changes in camera pose over time. Three CL-based approaches are introduced: the naive CL approach, the reservoir rehearsal approach, and the hybrid approach combining reservoir sampling with new data evaluation. The naive CL approach suffers from catastrophic forgetting, while the reservoir rehearsal approach mitigates this issue by sampling uniformly from past data. The hybrid approach further enhances performance by incorporating reservoir sampling and assessing new data for novelty. Experiments conducted in simulated and real-world environments demonstrate that the CL-based approaches, except for the naive approach, achieve competitive performance compared to traditional batch learning-based methods. This suggests that treating hand–eye calibration as a time sequence problem enables the extension of the learned space without complete retraining. The adaptability of the CL-based approaches facilitates accommodating changes in camera pose, leading to an improved hand–eye calibration system.

Real rain removal is a complex and challenging problem on real rain images. Although rain removal using the CycleGAN (Tang et al. in ECCV 2018 Workshop Proceedings, Part V [1]) was found to be superior as compared to other existing techniques, there are two aspects that can be improved on: (1) Preserve original scene details, and (2) Remove low-frequency rain distortion. Hence, this paper proposes a multi-scale representation technique, called the MS-CycleGANs, to address the remaining gaps of the CycleGAN in rain removal. It uses a pyramid framework that is made up of multi-scale CycleGANs. Such multi-scale representation is capable of removing rain distortions at multiple frequency sub-bands without sacrificing the scene details because the characteristics of both rain and no-rain domains can be learnt simultaneously by the proposed network at their frequency sub-bands. We compare the MS-CycleGANs to the CycleGAN on similar rain images for the networks’ training and testing. The results are compared quantitatively using the NIQE metric (Mittal et al. in IEEE Signal Process Lett 22:209–212, [2]) in this paper as we do not have any prior information on real rain distortions. The results have demonstrated the MS-CycleGANs’ superiority in removing real rain distortions, both qualitatively by visual check and also quantitatively using the NIQE metric (Mittal et al. in IEEE Signal Process Lett 22:209–212, [2]).

In this paper, we tackle the challenge of actively attending to visual scenes using a foveated sensor. We introduce an end-to-end differentiable foveated active vision architecture that leverages a graph convolutional network to process foveated images, and a simple yet effective formulation for foveated image sampling. Our model learns to iteratively attend to regions of the image relevant for classification. We conduct detailed experiments on a variety of image datasets, comparing the performance of our method with previous approaches to foveated vision while measuring how the impact of different choices, such as the degree of foveation, and the number of fixations the network performs, affect object recognition performance. We find that our model outperforms a state-of-the-art CNN and foveated vision architectures of comparable parameters and a given pixel or computation budget

Although many techniques are devised in rain removal research recently, using either the Convolutional Neural Network (CNN) [1] or the Generative Adversarial Network (GAN) [2], they always require both rain and no-rain training images exist in pairs to train their networks. Recently, the Cycle-Consistent Adversarial Networks (CycleGAN) [3] has demonstrated successful results in removing real rain distortion, without the no-rain training images [4]. Building on this success, we propose a new technique called the Wavelet-CycleGANs (W-CycleGANs) that has the same advantage. In addition, the wavelet properties can be used for the CycleGAN to remove rain from images at their frequency sub-bands, in the Hue, Saturation and Value (HSV) color space [5]. We train and compare the W-CycleGANs’ to the CycleGAN’s performance fairly, using the same set of rain images as the CycleGAN [4]. Their quantitative results are compared using the Natural Image Quality Evaluator (NIQE) [6] as real rain images in pairs simply do not exist in the real world. In addition, their qualitative results are compared using visual check at zoomed-in regions. Both results have demonstrated the W-CycleGANs’ superiority in removing real rain distortions.

Analogue film restoration, both for still photographs and motion picture emulsions, is a slow and laborious manual process. Artifacts such as dust and scratches are random in shape, size, and location; additionally, the overall degree of damage varies between different frames. We address this less popular case of image restoration by training a U-Net model with a modified perceptual loss function. Along with the novel perceptual loss function used for training, we propose a more rigorous quantitative model evaluation approach which measures the overall degree of improvement in perceptual quality over our test set.

Foveated vision captures a visual scene at space-variant resolution. This makes the application of parameterized convolutions to foveated images difficult as they do not have a dense-grid representation in cartesian space. Log-polar space is frequently used to create a dense grid representation of foveated images, however this image representation may not be appropriate for all applications. In this paper we rephrase the convolution operation as the Monte-Carlo estimation of the filter response of the foveated image and a continuous filter kernel, an idea that has seen frequent use for deep learning on point clouds. We subsume our convolution operation into a simple CNN architecture that processes foveated images in cartesian space. We evaluate our system in the context of image classification and show that our approach significantly outperforms an equivalent CNN processing a foveated image in log-polar space.

We present a hand-eye calibration approach based on a deep learning-based regression architecture to find the transformation between the robot end-effector and an external camera. For this, we hypothesise that it is possible to track a single reference point in the robot's end-effector to estimate the hand-eye geometric transformation using a deep neural network and a 3D vision system. To explore this hypothesis, we design three experiments to study the different components of our proposed network architecture while solving isolated cases of the hand-eye calibration problem. Our experimental results using a simulated environment show that our proposed approach has less than 1 mm error for translation and less than 2.31 degrees error for orientation. We also carried out experiments for our third approach in two real robotic testbeds (a Universal Robot 3 and the Rethink Baxter robot). Our approach achieves 2 mm and 5.9 degrees, 4.53 mm and 9.2 degrees of errors for the Universal Robot UR3 and the Rethink Baxter robot.

This work investigates the utility of a biologically motivated software retina model to pre-process and compress visual information prior to training and classification by means of a deep convolutional neural networks (CNNs) in the context of object recognition in robotics and egocentric perception. We captured a dataset of video clips in a standard office environment by means of a hand-held high-resolution digital camera using uncontrolled illumination. Individual video sequences for each of 20 objects were captured over the observable view hemisphere for each object and several sequences were captured per object to serve training and validation within an object recognition task. A key objective of this project is to investigate appropriate network architectures for processing retina transformed input images and in particular to determine the utility of spatio-temporal CNNs versus simple feed-forward CNNs. A number of different CNN architectures were devised and compared in their classification performance accordingly. The project demonstrated that the image classification task could be conducted with an accuracy exceeding 98

This paper proposes a novel automatically generating image masks method for the state-of-the-art Mask R-CNN deep learning method. The Mask R-CNN method achieves the best results in object detection until now, however, it is very time consuming and laborious to get the object Masks for training, the proposed method is composed by a two-stage design, to automatically generating image masks, the first stage implements a fully convolutional networks (FCN) based segmentation network, the second stage network, a Mask RCNN based object detection network, which is trained on the object image masks from FCN output, the original input image, and additional label information. Through experimentation, our proposed method can obtain the image masks automatically to train Mask R-CNN, and it can achieve very high classification accuracy with an over 90% mean of average precision (mAP) for segmentation.

Malignant Pleural Mesothelioma (MPM) is a rare cancer associated with exposure to asbestos fibres. It grows in the pleural space surrounding the lungs, exhibiting an irregular shape with high surface-to-volume ratio. Reliable measurements are important to assessing treatment efficacy, however these tumour characteristics make manual measurements time consuming, and prone to intra- and inter-observer variation. Previously we described a fully automatic Convolutional Neural Network (CNN) for volumetric measurement of MPM in CT images, trained and evaluated by seven-fold cross validation on 123 CT datasets with expert manual annotations. The mean difference between the manual and automatic volume measurements was not significantly different from zero (27.2 cm ^3 ; p = 0.225 ), the 95 ^3 , and the mean Dice coefficient was 0.64. Previous studies have focused on images with known MPM, sometimes even focusing on the lung with known MPM. In this paper, we investigate the false positive detection rate in a large image set with no known MPM. For this, a cohort of 14,965 subjects from the National Lung Screening Trial (NLST) were analysed. The mean volume of “MPM” found in these images by the automated detector was 3.6 cm ^3 (compared with 547.2 cm ^3 for MPM positive subjects). A qualitative examination of the one hundred subjects with the largest probable false detection volumes found that none of them were normal: the majority contain hyperdense pathology, large regions of pleural effusion, or evidence of pleural thickening. One false positive was caused by liver masses. The next step will be to evaluate the automated measurement accuracy on an independent, unseen, multi-centre data set.

Malignant Pleural Mesothelioma (MPM) is a cancer associated with prior exposure to asbestos fibres. Unlike most tumours, which are roughly spherical, MPM grows like a rind surrounding the lung. This irregular shape poses significant clinical and technical challenges. Accurate tumour measurements are necessary to determine treatment efficacy, but manual segmentation is tedious, time-consuming and associated with high intra- and inter-observer variation. In addition, uncertainty is compounded by poor differentiation in the computed tomography (CT) image between MPM and other common features. We describe herein an internal validation of a fully automatic tool to generate volumetric segmentations of MPM tumours using a convolutional neural network (CNN). The system was trained using the first 123 CT volumetric datasets from a planned total of 403 scans. Each scan was manually segmented to provide the expert ground truth. Evaluation was by seven-fold cross validation on a subset of 80/123 datasets that have full volumetric segmentations. The mean volume of MPM tumour in these datasets is 405.1 cm(3) (standard deviation 271.5 cm(3)). Following three-dimensional binary closing of the manual annotations to improve inter-slice consistency, the mean volume difference between the manual and automatic measurements is 27.2 cm(3), which is not significantly different from zero difference (p = 0.225). The 95% limits of agreement between the manual and automated measurements are between -417 and +363 cm(3). The mean Dice overlap coefficient was 0.64, which is comparable with inter-observer measurements reported elsewhere. To our knowledge, this is the first algorithm of its kind that fully automates and evaluates measurement of the MPM tumour volume. The next step will be to evaluate the method on the remaining unseen multi-centre evaluation set. Such an algorithm has possible future application to pharmaceutical trials (where it offers a repeatable study end point) and to routine care (where it allows tumour progression to be assessed rapidly to enhance therapeutic clinical decision making).

Deep Reinforcement Learning (DRL) has gained much attention for solving robotic hand-eye coordination tasks from raw pixel values. Despite promising results, training agents using images is hardware intensive often requiring millions of training steps to converge incurring long training times and increased risk of wear and tear on the robot. To speed up training, images are often cropped and downscaled resulting in a smaller field of view and loss of valuable high-frequency data. In this paper, we propose training the vision system using supervised learning prior to training robotic actuation using Deep Deterministic Policy Gradient (DDPG). The vision system uses a software retina, based on the mammalian retino-cortical transform, to preprocess full-size images to compress image data while preserving the full field of view and high-frequency visual information around the fixation point prior to processing by a Deep Convolutional Neural Network (DCNN) to extract visual state information. Using the vision system to preprocess the environment improves the agent's sample complexity and network update speed leading to significantly faster training with reduced image data loss. Our method is used to train a DRL system to control a real Baxter robot's arm, processing full-size images captured by an in-wrist camera to locate an object on a table and centre the camera over it by actuating the robot arm.

Purpose The traditional methods have difficulty to inspection various types of copper strips defects as inclusions, pits and delamination defects under uneven illumination. Therefore, this paper aims to propose an optimal real Gabor filter model for inspection; however, improper selection of Gabor parameters will cause the boundary between the defect and the background image to be not very clear. This will make the defect and the background cannot be completely separated. Design/methodology/approach The authors proposed an optimal Real Gabor filter model for inspection of copper surface defects under uneven illumination. This proposed method only requires a single filter by calculating the specific convolution energy of the Gabor filter with the image. The Real Gabor filter’s parameter is optimized by particle swarm optimization (PSO), which objective fitness function is maximization of the Gabor filter’s energy average divided by the energy standard deviation, the objective makes a distinction between the defect and normal area. Findings The authors have verified the effect with different iterations of parameter optimization using PSO, the effects with different control constant of energy and neighborhood window size of real Gabor filter, the experimental results on a number of metal surface have shown the proposed method achieved a well performance in defect recognition of metal surface. Originality/value The authors propose a defect detection method based on particle swarm optimization for single Gabor filter parameters optimization. This proposed method only requires a single filter and finds the best parameters of the Gabor filter. By calculating the specific convolution energy of the Gabor filter and the image, to obtain the best Gabor filter parameters and to highlight the defects, the particle swarm optimization algorithm’s fitness objective function is maximize the Gabor filter's average energy divided by the energy standard deviation.

We present an investigation into adopting a model of the retino-cortical mapping, found in biological visual systems, to improve the efficiency of image analysis using Deep Convolutional Neural Nets (DCNNs) in the context of robot vision and egocentric perception systems. This work has now enabled DCNNs to process input images approaching one million pixels in size, in real time, using only consumer grade graphics processor (GPU) hardware in a single pass of the DCNN.

This paper addresses the problem of removing rain disruption from images for outdoor vision systems. The Cycle-Consistent Generative Adversarial Network (CycleGAN) is proposed as a more promising rain removal algorithm, as compared to the state-of-the-art Image De-raining Conditional Generative Adversarial Network (ID-CGAN). The CycleGAN has an advantage in its ability to learn the underlying relationship between the rain and rain-free domain without the need of paired domain examples. Based on rain physical properties and its various phenomena, five broad categories of real rain distortions are proposed in this paper. For a fair comparison, both networks were trained on the same set of synthesized rain-and-ground-truth image-pairs provided by the ID-CGAN work, and subsequently tested on real rain images which fall broadly under these five categories. The comparison results demonstrated that the CycleGAN is superior in removing real rain distortions.

We present a robot vision approach to deformable object classification, with direct application to autonomous service robots. Our approach is based on the assumption that continuous perception provides robots with greater visual competence for deformable objects interpretation and classification. Our approach classifies the category of clothing items by continuously perceiving the dynamic interactions of the garment’s material and shape as it is being picked up. For this, we extract continuously visual features of a RGB-D video sequence and we fuse features by means of the Locality Constrained Group Sparse Representation (LGSR) algorithm. To evaluate the performance of our approach, we created a fully annotated database featuring 150 garment videos in random configurations. Experiments demonstrate that by continuously observing an object deform, our approach achieves a classification score of 66.7%, outperforming state-of-the-art approaches by a ∼ 27 . 3 % increase.

The following topics are dealt with: mobile robots; control system synthesis; learning (artificial intelligence); feature extraction; robot vision; autonomous aerial vehicles; feedback; feedforward neural nets; nonlinear control systems; multi-robot systems.

This paper presents a novel robot vision architecture for perceiving generic 3-D clothes configurations. Our architecture is hierarchically structured, starting from low-level curvature features to mid-level geometric shapes and topology descriptions, and finally, high-level semantic surface descriptions. We demonstrate our robot vision architecture in a customized dual-arm industrial robot with our inhouse developed stereo vision system, carrying out autonomous grasping and dual-arm flattening. The experimental results show the effectiveness of the proposed dual-arm flattening using the stereo vision system compared with the single-arm flattening using the widely cited Kinect-like sensor as the baseline. In addition, the proposed grasping approach achieves satisfactory performance when grasping various kind of garments, verifying the capability of the proposed visual perception architecture to be adapted to more than one clothing manipulation tasks.