In action recognition, many different network structures of spatiotemporal features extractions has been proposed, and performed well on several mainstream datasets. Inevitably, one question occurs to us: is there any transferable characterizes between different models? In this paper, we discuss such problem by introducing a cross-architecture transferring learning scheme, dubbed soft transferring learning, aiming to overcome the limitation of divergence of different network structures. A multi-stage semi-supervision training procedure is conducted to keep the consistency internally between two different models from bottom to top. To this end, we introduce two kinds of cross-structure metric strategy to compute the mismatch value in different features level, together with entropy classification loss, integrated to be a three-stage supervision method. We additionally design a new network to learn from the supervisors, which have been trained on large-scale datasets. We fine-tune supervision model and train our new model on UCF101 and HMDB51 datasets, experiment results demonstrate the feasibility of soft transferring method, extend transfer learning to a broader sense, and show the flexibility of deploying of existing models. Our method is designed easily generalized to different networks in other computer vision task.
Proposed in 1991, Least Mean Square Error Reconstruction for self-organizing network, shortly Lmser, was a further development of the traditional auto-encoder (AE) by folding the architecture with respect to the central coding layer and thus leading to the features of Duality in Connection Weight (DCW) and Duality in Paired Neurons (DPN), as well as jointly supervised and unsupervised learning which is called Duality in Supervision Paradigm (DSP). However, its advantages were only demonstrated in a one-hidden-layer implementation due to the lack of computing resources and big data at that time. In this paper, we revisit Lmser from the perspective of deep learning, develop Lmser network based on multiple fully-connected layers, and confirm several Lmser functions with experiments on image recognition, reconstruction, association recall, and so on. Experiments demonstrate that Lmser indeed works as indicated in the original paper, and it has promising performance in various applications.
Audio source separation is an important but challenging problem for many applications due to the only available single channel mixed signal. This work proposes a novel Non-Local Multi-scale Multi-band DenseNet model termed as NLMMDenseNet for audio source separation by jointly exploring the long-term dependencies and recovering the missing information around bands’ borders. Specifically, to well leverage the long-term dependencies among the audio spectrogram, we propose a new non-local model by incorporating the non-local layer into MMDenseNet. It enables the proposed model to capture different audio sources features. Besides, the proposed model can also capture cross-band features, which are used to recover the missing information around bands’ borders. The proposed model outperforms state-of-the-art results on the widely-used MIR-1K and DSD100 datasets by taking advantages of global information and bands’ border information.
Deep bidirectional Intelligence (BI) via YIng YAng (IA) system, or shortly Deep IA-BI, is featured by circling A-mapping and I-mapping (or shortly AI circling) that sequentially performs each of five actions. A basic foundation of IA-BI is bidirectional learning that makes the cascading of A-mapping and I-mapping (shortly A-I cascading) approximate an identical mapping, with a nature of layered, topology-preserved, and modularised development. One exemplar is Lmser that improves autoencoder by incremental bidirectional layered development of cognition, featured by two dual natures DPN and DCW. Two typical IA-BI scenarios are further addressed. One considers bidirectional cognition and image thinking, together with a proposal that combines theories of Hubel-Wiesel's versus Chen's. The other considers bidirectional integration of cognition, knowledge accumulation, and abstract thinking for improving implementation of searching, optimising, and reasoning. Particularly, an IA-DSM scheme is proposed for solving a doubly stochastic matrix (DSM) featured combinatorial tasks such as travelling salesman problem, and also a Subtree driven reasoning scheme is proposed for improving production rule based reasoning. In addition, some remarks are made on relations of Deep IA-BI to Hubel and Wiesel theory, Sperry theory, and A5 problem solving paradigm.
This paper presents an object detection method that can simultaneously estimate the positions and depth of the objects from multiplexed images. Multiplexed image is produced by a new type of imaging device that collects the light from different fields of view using a single image sensor, which is originally designed for stereo, 3D reconstruction and broad view generation using computational imaging. Intuitively, multiplexed image is a blended result of the images of multiple views and both of the appearance and disparities of objects are encoded in a single image implicitly, which provides the possibility for reliable object detection and depth/disparity estimation. Motivated by the recent success of CNN based detector, a multi-anchor detector method is proposed, which detects all the views of the same object as a clique and uses the disparity of different views to estimate the depth of the object. The proposed method is interesting in the following aspects: firstly, both locations and depth of the objects can be simultaneously estimated from a single multiplexed image; secondly, there is almost no computation load increase comparing with the popular object detectors; thirdly, even in the blended multiplexed images, the detection and depth estimation results are very competitive. There is no public multiplexed image dataset yet, therefore the evaluation is based on the simulated multiplexed image using the stereo images from KITTI, and very encouraging results have been obtained.
In this paper, a nonlocal low rank prior associated with spatial smoothness and spectral collaborative sparsity are integrated together for unmixing the hyperspectral data. Based on a fact that the hyperspectral images have self-similarity in nonlocal sense and smoothness in the local sense. To explore the spatial self-similarity, the nonlocal cubic patches are grouped together to form a low-rank matrix. Then, in the framework of linear mixture model, the nuclear norm is constrained to the abundance matrix of these similar patches to enforce low-rank property. In addition, the spectral and local spatial information is also taken into account by introducing collaborative sparsity and TV regularization terms, respectively. Finally, the proposed method is tested on two simulated data sets and a real data set and the results show that the proposed algorithm produces better performance than other state-of-the-art algorithms.
Person re-identification (re-ID) is a challenging task since the same person captured by different cameras can appear very differently, due to the uncontrolled factors such as occlusion, illumination, viewpoint and pose variation etc. Attention-based person re-ID methods have been extensively studied to focus on discriminative regions of the last convolutional layer, which, however, ignore the low-level fine-grained information. In this paper, we propose a novel SliceNet with efficient feature augmentation modules for open-world person re-identification. Specifically, with the philosophy of divide and conquer, we divide the baseline network into three sub-networks from low, middle and high levels, which are called slice networks, followed by a Self-Alignment Attention Module respectively to learn multi-level discriminative parts. In contrast with existing works that uniformly partition the images into multiple patches, our attention module aims to learn self-alignment masks for discovering and exploiting the align-attention regions. Further, SliceNet is combined with the attention free baseline network to characterize global features. Extensive experiments on the benchmark datasets including Market-1501, CUHK03, and DukeMTMC-reID show that our proposed SliceNet achieves favorable performance compared with the state-of-the art methods.
Linear non-Gaussian Acyclic Model (LiNGAM) is a well-known model for causal discovery from observational data. Existing estimation methods are usually based on infinite sample theory and often fail to obtain an ideal result in the small samples. However, it is commonplace to encounter non-Gaussian data with small or medium sample sizes in practice. In this paper, we propose a Minimal Set-based LiNGAM algorithm (MiS-LiNGAM) to address the LiNGAM with small samples. MiSLiNGAM is a two-phase and greedy search algorithm. Specifically, in the first phase, we find the skeleton of the network using the regression-based conditional independence test, which helps us reduce the complexity in finding the minimal LiNGAM set of the second phase. Further, this independence test we applied guarantees the reliability when the number of conditioning variables increases. In the second phase, we give an efficient method to iteratively select the minimal LiNGAM set with the skeleton and learn the causal network. We also present the corresponding theoretical derivation. The experimental results on simulated networks and real networks are presented to demonstrate the efficacy of our method.
Facial landmark detection is one of the most important tasks in face image and video analysis. Existing algorithms based on deep convolutional neural networks have achieved good performance in public benchmarks and practical applications such as face verification, expression analysis, beauty applications and so on. However, the performance of a facial landmark detector degrades significantly when dealing with challenging facial images in the presence of extreme appearance variations such as pose, expression, occlusion, etc. To mitigate these difficulties, we propose a robust facial landmark detection algorithm based on coordinates regression in an end-to-end training fashion. By using the soft-argmax function, the network weights can be optimised with a mixed loss function. The online pose-based data augmentation technology is used to effectively solve the data imbalance problem and improve the robustness of the proposed method. Experiments conducted on the 300-W and AFLW datasets demonstrate that the performance of the proposed algorithm is competitive to the state-of-the-art heatmap regression algorithms, in terms of accuracy. Besides, our method achieves real-time speed on 300-W with 68 landmarks, which runs at 85 FPS on a Tesla v100 GPU.
Recently, object detection has made significant progress due to the development of deep learning. Since the traffic lights are extremely small objects, it leads to unsatisfactory performance when directly applying the off-the-shelf methods based on deep convolutional neural networks. To deal with this problem, we propose an improved detection network based on Faster R-CNN framework. By introducing an attention module on the top of the network, the network can focus better on the small object regions. At the same time, the features from shallow layers are leveraged for classification and bounding box regression, in which the features of small objects can be captured better. In addition, we design a two-branch network for detecting the traffic light box and the bulb box at the same time. In this manner, the performance of traffic light detection is improved obviously. Compared with other detection algorithms, our model achieves competitive results on VIVA traffic light challenge dataset.
To deal with the emergency pre-plans saved by the power grid dispatch department, so that the dispatcher can quickly retrieve and match similar accidents in the pre-plans, then they can learn from the experience of previous relevant situations, it is necessary to extract the information of the pre-plans and extract its key information. Therefore, deep learning method with strong generalization ability and learning ability and continuous improvement of model can be adopted. However, this method usually requires a large amount of data, but the existing labeling data in the power grid field is limited and the manual method for data labeling is a huge workload. Therefore, in the case of insufficient data, this paper aims to solve how to use deep learning method for effective information extraction? This paper modifies the ULMFiT model and uses it to carry out word vector training, adopting transfer learning method to introduce annotating datasets in the open field and combining with the data in the field of power grid to training model. In this way, the semantic relation of power grid domain is introduced into the syntactic analysis of the pre-plans, and we can further complete the information extraction. Experimental verification is carried out in this paper, the results show that, in the case of insufficient corpus or small amount of annotated data, this method can solve the problem of part of speech analysis errors, it can also improve the accuracy of syntactic analysis, and the experimental verifies the effectiveness of this method.
We propose an unsupervised novel method, Attention-Pixel and Attention-Channel Network (APAC-Net), for unsupervised monocular learning of estimating scene depth and ego-motion. Our model only utilizes monocular image sequences and does not need additional sensor information, such as IMU and GPS, for supervising. The attention mechanism is employed in APAC-Net to improve the networks’ efficiency. Specifically, three attention modules are proposed to adjust feature weights when training. Moreover, to minimum the effect of noise, which is produced in the reconstruction processing, the Image-reconstruction loss based on PSNR $$L_{PSNR}$$ is used to evaluation the reconstruction quality. In addition, due to the fail depth estimation of the objects closed to camera, the Temporal-consistency loss $$L_{Temp}$$ between adjacent frames and the Scale-based loss $$L_{Scale}$$ among different scales are proposed. Experimental results showed APAC-Net can perform well in both the depth and ego-motion tasks, and it even behaved better in several items on KITTI and Cityscapes.
Controllability of networks widely existing in real-life systems have been a critical and attractive research subject for both network science and control systems communities. Research in network controllability has mostly focused on the effects of the network structure on its controllability, and some studies have begun to investigate the controllability robustness of complex networks. Cascading failure is common phenomenon in many infrastructure networks, which largely affect normal operation of networks, and sometimes even lead to collapse, resulting in considerable economic losses. The robustness of network controllability against the cascading failure is studied by a linear load-capacity model with a breakdown probability in this paper. The controllability of canonical model networks under different node attack strategies is investigated, random failure and malicious attack. It is shown by numerical simulations that the tolerant parameter of load-capacity model has an important role in the emergence of cascading failure, independent to the types of network. The networks with moderate average degree are more vulnerable to the cascading failure while these with high average degree are very robust. In particular, betweenness attack strategy is more harmful to the network controllability than degree attack one, especially for the scale-free networks.
Few-shot learning aims to learn a model which can quickly generalize with only a small number of labeled samples per class. The situation we consider is how to use the information of the test set to generate the better prototype representation of the training set. In this paper, based on attention mechanism we propose a flexible and efficient framework for few-shot feature fusion, called Attention Relational Network (ARN) which is a three-branch structure of embedding module, weight module and matching module. Specifically, with attention mechanism, the proposed ARN can model adaptively the constribution weights of sample features from embedding module and then generate the prototype representations by weighted fusion of the sample features. Finally, the matching module identify target sample by calculating the matching scores. We evaluated this method on the MiniImageNet and Omniglot dataset, and the experiment proved that our method is very attractive.
Weakly-supervised semantic segmentation with image-level labels is a important task as it directly associates high-level semantic to low-level appearance, which can significantly reduce human efforts. Despite the remarkable progress, it is still not as good as fully supervised segmentation methods. To improve the accuracy, in this paper, we proposed a novel framework of weakly-supervised semantic segmentation with mean teacher (WSSS-MT) learning to advance the class estimation of image pixels. More specifically, our proposed framework includes a student network and a teacher network in the segmentation module, which aims to effectively utilize information of the training process. The student learns the semantic segmentation network with an updated supervision, while the teacher uses the exponential moving average of the student to achieve a more accurate estimation of supervision. WSSS-MT employs the trained teacher as final segmentation network. Experimental results on the PASCAL VOC 2012 dataset show that the performance of our framework is better than the competing methods.
Vehicle speed analysis based on the video is a challenging task in the field of traffic safety, which has high requirements for accuracy and computational burden. The drone’s video is taken from a top-down perspective, providing more complete view comparing to the common surveillance cameras in poles. In this paper, we introduce a Gaussian Filter to deal with the estimated speed data which are extracted by a multiple objects tracking method composed of You Only Look Once (YOLOv3) and Kalman Filter. We exploit the capability of Gaussian Filter to suppress data noise appearing in the process of tracking vehicles from drone videos, and thus use the filter to solve the case where the estimated vehicle speed is fluctuated along the ongoing direction. On the other hand, we built a vehicle dataset from the drone’s videos we mentioned above which additionally contains vehicle’s real speed information. Experimental results showed that our method is effective to improve the accuracy of vehicle speed estimated by our tracking module. It can improve Mean Squared error (MSE) accuracy 80.5% on experimental data.
Precise segmentation of key tissues in medical images is of great significance. Although deep neural networks have achieved promising results in many medical image segmentation tasks, it is still a challenge for volumetric medical image segmentation due to the limited computing resources and annotated datasets. In this paper, we propose a multi-resolution coarse-to-fine segmentation framework to perform accurate segmentation. The proposed framework contains a coarse stage and a fine stage. The coarse stage with low-resolution data provide high semantic cues for the fine stage. Moreover, we embed active learning processes into coarse-to-fine framework for sparse annotation, the proposed multiple query criteria active learning methods can select high-value slices to label. We evaluated the effectiveness of proposed framework on two public brain MRI datasets. Our coarse-to-fine networks outperform other competitive methods under the condition of fully supervised training. In addition, the proposed active learning method only need 30% to 40% slices of one scan to produce relatively better dense prediction results than non-active learning method and one query criteria active learning methods.
Text classification is a basic task in natural language processing. Recently, pre-training models such as BERT have achieved outstanding results compared with previous methods. However, BERT fails to take into account local information in the text such as a sentence and a phrase. In this paper, we present a BERT-CNN model for text classification. By adding CNN to the task-specific layers of BERT model, our model can get the information of important fragments in the text. In addition, we input the local representation along with the output of the BERT into the transformer encoder in order to take advantage of the self-attention mechanism and finally get the representation of the whole text through transformer layer. Extensive experiments demonstrate that our model obtains competitive performance against state-of-the-art baselines on four benchmark datasets.
With the increasing number of machine learning parameters, the requirements on data quantity are getting higher and higher to train a good model. The choice of methods and the optimization of parameters can improve the model while the quality and quantity of the data determine the upper limit of the model. However, in realistic scenarios, it is quite challenging to get a lot of tag data. Therefore, it is natural to realize data augmentation by transforming the original data. We use three methods for data augmentation on different scales of original data in solving the crime prediction problem based on the description of the cases, and find that the effects of data augmentation are different for different models and different fundamental data quantities.
U-Net has been one of the important deep learning models applied for biomedical image segmentation for a few years. In this paper, inspired by the way how fully convolutional network (FCN) makes dense predictions, we modify U-Net by adding a new bypass for the expansive path. Before combining the contracting path with the upsampled output, we connect with the feature maps from a deeper encoding convolutional layer for the decoding up-convolutional units, and sum up the information learned from both sides. Also, we have implemented this modification to recurrent residual convolutional neural network based on U-Net as well. The experimental results show that the proposed bypass-based U-Net can gain further context information, especially the details from the previous convolutional layer, and outperforms the original U-Net on the DRIVE dataset for retinal vessel segmentation and the ISBI 2018 challenge for skin lesion segmentation.
