Viewers of 360-degree videos are provided with both visual modality to characterize their surrounding views and audio modality to indicate the sound direction. Though both modalities are important for saliency prediction, little work has been done by jointly exploiting them, which is mainly due to the lack of audio-visual saliency datasets and insufficient exploitation of the multi-modality. In this article, we first construct an audio-visual saliency dataset with 57 360-degree videos watched by 63 viewers. Through a deep analysis of the constructed dataset, we find that the human gaze can be attracted by the auditory cues, resulting in a more concentrated saliency map if the sound source's location is further provided. To jointly exploit the visual and audio features and their correlation, we further design a saliency prediction network for 360-degree videos (SVGC-AVA) based on spherical vector-based graph convolution and audio-visual attention. The proposed spherical vector-based graph convolution can process visual and audio features directly in the sphere domain, thus avoiding projection distortion incurred by traditional CNN-based predictors. In addition, the audio-visual attention scheme explores self-modal and cross-modal correlation for both modalities, which are further hierarchically processed with the U-Net's multi-scale structure of SVGC-AVA. Evaluations on both our and public datasets validate that SVGC-AVA can achieve higher prediction accuracy, both qualitatively and subjectively.

Predicting the saliency map of a 360-degree video is the key for various downstream tasks, such as saliency-based compression and tile-based adaptive streaming. Besides static salient objects, the moving target will also contribute to the saliency map. Therefore, the joint exploitation of spherical spatio-temporal information is necessary for an accurate saliency prediction. The spherical spatial feature extraction, however, is hindered by the non-Euclidean geometric nature of spherical data, which imposes difficulty on direct extraction of the spatial features with traditional convolutional neural networks (CNNs). While the efficient exploitation of temporal correlation between these spherical spatial features remains another challenge, which requires the extraction of spherical optical flows for explicit motion information. To address these, in this paper, we first propose a spherical graph-based Farneback algorithm to extract the spherical optical flows directly in the sphere domain, by leveraging the GICOPix uniform sampling scheme. We then design a 3D separable graph convolutional network-based saliency prediction framework, named 360Spred, by taking both the spherical frames and spherical optical flows as input. The proposed 360Spred framework is based on the U-Net structure, with a 3D separable graph convolution (3DSGC) operator that directly extracts the visual and motion features in the sphere domain and exploits temporal correlation of both the high-level and low-level spatial features. Experimental results on two public datasets show that 360Spred can achieve a better performance than other baseline models in terms of the saliency prediction accuracy for 360-degree videos.

Learning a rotation-invariant (RI) representation is of significant importance for real-world point cloud segmentation that is perturbed by arbitrary rotations. Recent principal component analysis (PCA)-based methods provide an effective alternative to align point clouds and produce the RI representation under the preservation of global information. However, conventional PCA with 3-D coordinates cannot fully represent high-dimensional geometric structures like surfaces and curves and cannot uniformly align these structures for learning RI representation. In this paper, we propose a novel rotation-invariant method for point cloud segmentation, which leverages kernel PCA (KPCA) for aligning point clouds in a projected high-dimensional space via non-linear mapping and develops a Geometry-based Weighted Convolution (GWConv) to distinguish part boundaries during segmentation. Specifically, the KPCA produces a RI representation with polynomial kernels for effectively representing complicated geometric structures in point clouds. Moreover, the GWConv incorporates geometric structures into convolution and enhances neighboring points with similar geometry for fine-grained segmentation based on the RI representation. Experimental results demonstrate that the proposed method can achieve competitive performance with the state-of-the-arts and outperforms existing PCA-based methods in part segmentation on ShapeNet. Furthermore, it achieves evident performance gains on complicated 3-D shapes such as Earphone and Car and facilitates segmentation around the part boundaries.

Graph convolutional networks (GCNs) are widely adopted for spherical data processing, striking a balance between rotation equivariance and computation efficiency. However, current GCN-based approaches either apply radial filters with a fixed convolution kernel, or adopt attention schemes to learn a variable kernel based only on node features, which lack expressiveness and may suffer over-smoothing problem. To address this, in this paper, we propose a global-local attention-based spherical graph convolution (GlasGC) for spherical data representation. Specifically, the designed graph convolution includes a structure-enhanced local attention module to estimate the structure correlation efficiently based on topological characteristics of the given spherical graph. The structure correlation is then incorporated with the feature correlation to form the local attention that determines importance of a neighbor node to the center node. We also design a global attention module to obtain the importance scores of different nodes based on both node features and graph topology, which acts as a message-passing enhancer for informative nodes and can relieve the over-smoothing problem. Based on our proposed GlasGC, we design specific GlasGCNs for classification and semantic segmentation of spherical data. Empirical evaluations on these tasks demonstrate that our proposed GlasGCNs can generally achieve better classification and segmentation performance.

Objective.Noise-assisted multivariate empirical mode decomposition (NA-MEMD) based causal decomposition depicts a cause and effect relationship that is not based on the term of prediction, but rather on the phase dependence of time series. Here, we present the NA-MEMD based causal decomposition approach according to the covariation and power views traced to Hume and Kant:a prioricause-effect interaction is first acquired, and the presence of a candidate cause and of the effect is then computed from the sensory input somehow.Approach.Based on the definition of NA-MEMD based causal decomposition, we show such causal relation is a phase relation where the candidate causes are not merely followed by effects, but rather produce effects.Main results.The predominant methods used in neuroscience (Granger causality, empirical mode decomposition-based causal decomposition) are validated, showing the applicability of NA-MEMD based causal decomposition, particular to brain physiological processes in bivariate and multiscale time series.Significance.We point to the potential use in the causality inference analysis in a complex dynamic process.

This paper investigates the association between consecutive ambient air pollution and Chronic Obstructive Pulmonary Disease (COPD) hospitalization in Chengdu China. The three-year (2015-2017) time series data for both ambient air pollutant concentrations and COPD hospitalizations in Chengdu are approved for the study. The big data statistic analysis shows that Air Quality Index (AQI) exceeded the lighted air polluted level in Chengdu region are mainly attributed to particulate matters (i.e., PM 2.5 and PM 10 ). The time series study for consecutive ambient air pollutant concentrations reveal that AQI, PM 2.5 , and PM 10 are significantly positive correlated, especially when the number of consecutive polluted days is greater than nine days. The daily COPD hospitalizations for every 10 μg/m 3 increase in PM 2.5 and PM 10 indicate that consecutive ambient air pollution can lead to an appearance of an elevation of COPD admissions, and also present that dynamic responses before and after the peak admission are different. Support Vector Regression (SVR) is then used to describe the dynamics of COPD hospitalizations to consecutive ambient air pollution. These findings will be further developed for region specific, hospital early notifications of COPD in responses to consecutive ambient air pollution.

In this article, we study the server-side rate adaptation problem for streaming tile-based adaptive 360-degree videos to multiple users who are competing for transmission resources at the network bottleneck. Specifically, we develop a convolutional neural network (CNN)-based viewpoint prediction model to capture the nonlinear relationship between the future and historical viewpoints. A Laplace distribution model is utilized to characterize the probability distribution of the prediction error. Given the predicted viewpoint, we then map the viewport in the spherical space into its corresponding planar projection in the 2-D plane, and further derive the visibility probability of each tile based on the planar projection and the prediction error probability. According to the visibility probability, tiles are classified as viewport, marginal and invisible tiles. The server-side tile rate allocation problem for multiple users is then formulated as a non-linear discrete optimization problem to minimize the overall received video distortion of all users and the quality difference between the viewport and marginal tiles of each user, subject to the transmission capacity constraints and users' specific viewport requirements. We develop a steepest descent algorithm to solve this non-linear discrete optimization problem, by initializing the feasible starting point in accordance with the optimal solution of its continuous relaxation. Extensive experimental results show that the proposed algorithm can achieve a near-optimal solution, and outperforms the existing rate adaptation schemes for tile-based adaptive 360-video streaming.

Causal decomposition depicts a cause-effect relationship that is not based on the concept of prediction, but based on the phase dependence of time series. It has been validated in both stochastic and deterministic systems and is now anticipated for its application in the complex dynamic process. Here, we present an extension of causal decomposition in the mutual complex dynamic process: cause and effect of time series are inherited in the decomposition of intrinsic components in a similar time scale. Furthermore, we illustrate comparative studies with predominate methods used in neuroscience, and show the applicability of the method particularly to physiological time series in brain-muscle interactions, implying the potential to the causality analysis in the complex physiological process.

Convolutional neural networks (CNNs) designed for low-dimensional regular grids will unfortunately lead to non-optimal solutions for analyzing spherical images, due to their different geometrical properties from planar images. In this paper, we generalize the grid-based CNNs to a non-Euclidean space by taking into account the geometry of spherical surfaces and propose a Spherical Graph Convolutional Network (SGCN) to encode rotation equivariant representations. Specifically, we propose a spherical graph construction criterion showing that a graph needs to be regular by evenly covering the spherical surfaces in order to design a rotation equivariant graph convolutional layer. For the practical case where the perfectly regular graph does not exist, we design two quantitative measures to evaluate the degree of irregularity for a spherical graph. The Geodesic ICOsahedral Pixelation (GICOPix) is adopted to construct spherical graphs with the minimum degree of irregularity compared to the current popular pixelation schemes. In addition, we design a hierarchical pooling layer to keep the rotation-equivariance, followed by a transition layer to enforce the invariance to the rotations for spherical image classification. We evaluate the proposed graph convolutional layers with different pixelations schemes in terms of equivariance errors. We also assess the effectiveness of the proposed SGCN in fulfilling rotation-invariance by the invariance error of the transition layers and recognizing the spherical images and 3D objects.

Sending only the viewport of interest provides a solution for 360-degree video streaming under the current bandwidth-constrained infrastructure. To this end, the user viewport requires to be prefetched in advance by conducting viewport prediction. To more accurately capture the nonlinear and long-term dependent relation between the future and past viewports, we develop a single viewport prediction model using convolutional neural network (CNN), in which the pooling layers are dropped and more convolutional layers are added for stronger nonlinear fitting ability. Further, we design a viewport trajectory prediction model based on recurrent neural network (RNN) which learns long-term dependency in sequential viewports. Specially, it is capable to estimate future viewport trajectory and support variable-size prediction window with low complexity. Finally, a correlation filter-based viewport tracker (CFVT) is proposed to perform content-aware viewport prediction. The combination of the RNN and the CFVT through a fusion model enables them to complement each other which is validated by significant improvement in prediction accuracy.

This study proposed a detection approach for the congestive heart failure (CHF) by short-time electrocardiographic monitoring. Recent literature only reported that RR intervals and Heart Rate Variability (HRV) indicated key hidden information to discriminate CHF groups from healthy controls. However whether it was possible to find certain short-time electrocardiographic monitoring duration for CHF clinical diagnoses, has not been well addressed. In the study, databases of 54 normal subjects and 15 CHF patients from PhysioNet were introduced. Signals were classified into variable assessment lengths. Based on R-R intervals in the assessment length, raw R-R intervals, mean and standard deviation (STD) of R-R intervals, and clinically standard features of shortterm (5-min) Heart Rate Variability (HRV), were comparatively analyzed, while combining with classifiers of Recurrent Neural Network (RNN), Random Forest (RF), and Support Vector Machine (SVM). The Leave-one-out Cross-Validation (LOOCV) was adopted for performance verification, by which the model extracted from certain assessment length was utilized to test measured data of a subject with the same length. Results showed that based on testing databases, a specific 30-minute duration can be achieved by choosing HRV features in full with sensitivity of 88.55% and specificity of 94.81%. It was believed that a short-time electrocardiographic monitoring for the CHF detection could be feasible if standard HRV features together with the classifier of RF or RNN are adopted. It implied that a short-time electrocardiographic monitoring can be applied for fast reference advice of CHF in urgent medical conditions.

In order to offer a reliable, fast, and offset-free tracking performance for the regulation of heart rate (HR) during treadmill exercise, a two-input single-output (2ISO) control system by simultaneously manipulating both treadmill speed and gradient is proposed. The decentralized integral controllability (DIC) analysis is extended to nonlinear and non-square processes especially for a 2ISO process, namely multi-loop integral controllability (MIC). The proposed multi-loop integral control-based HR regulation by manipulating treadmill speed and gradient is then validated through a comparative treadmill experiment that compares the system performance of the proposed 2ISO MIC control loop with that of single-input single-output (SISO) loops, speed/gradient-to-HR. The experimental validation presents that by simultaneously using two control inputs, the automated system can achieve the fastest HR tracking performance and stay close to the reference HR during steady state, while comparing with two SISO structures, and offer the fault-tolerant ability if the gains of the two multi-loop integral controllers are well tuned. It has a vital implication for the applications of exercise rehabilitation and fitness in relation to the automated control system.

How to balance the tradeoff between the user experience and bandwidth utilization emerges a critical challenge for multiuser 360-degree video adaptive streaming. This paper studies the server-side rate adaptation strategy for multiple users which are competing for the server bandwidth capacity. A tile visibility probability model is established, by which the tiles are classified into predicted, marginal and invisible types. A fine-grained rate adaptation problem is formulated as a nonlinear integer programming (NIP) problem, which aims at maximizing the video quality and navigation smoothness for multiple users. Thereafter, a steepest ascent algorithm with feasible starting point is developed to solve the proposed NIP problem in polynomial time. Finally, simulation results verify the performance of the proposed rate adaptation strategy.

The use of the Empirical Mode Decomposition (EMD) for nonlinear signal processing has been popularized in the recent years. However, its utility for the processing of multichannel Electromyography (EMG) signals is still limited. This paper investigates the decomposition performance of multichannel EMGs by using the EMD-based approaches, Ensemble EMD (EEMD), Multivariate EMD (MEMD), and Noise-Assisted MEMD (NA-MEMD). In the experiment, 11 male subjects undergo three exercise programs, leg extension from a sitting position, flexion of the leg up, and gait, while electrodes are placed on the muscle groups, biceps femoris, vastus medialis, rectus femoris, and semitendinosus. The outcomes are then quantitatively estimated on the basis of three criterions, the number of Intrinsic Mode Functions (IMFs), mode-alignment and mode-mixing. Results show both MEMD and NA-MEMD can guarantee equal numbers of IMFs, whereas for mode-alignment and mode-mixing, NA-MEMD is optimal compared with MEMD and EEMD, and MEMD is merely better than EEMD. This finding implies that NA-MEMD is effective for simultaneously analyzing IMFs based frequency bands. It has a vital clinical implication in exploring the neuromuscular patterns that enable the multiple muscle groups to coordinate while performing functional activities of daily living.

Motor imagery is an experimental paradigm implemented in cognitive neuroscience and cognitive psychology. To investigate the asymmetry of the strength of cortical functional activity due to different single-hand motor imageries, functional magnetic resonance imaging (fMRI) data from right handed normal subjects were recorded and analyzed during both left-hand and right-hand motor imagery processes. Then the average power of blood oxygenation level-dependent (BOLD) signals in temporal domain was calculated using the developed tool that combines Welch power spectrum and the integral of power spectrum approach of BOLD signal changes during motor imagery. Power change analysis results indicated that cortical activity exhibited a stronger power in the precentral gyrus and medial frontal gyrus with left-hand motor imagery tasks compared with that from right-hand motor imagery tasks. These observations suggest that right handed normal subjects mobilize more cortical nerve cells for left-hand motor imagery. Our findings also suggest that the approach based on power differences of BOLD signals is a suitable quantitative analysis tool for quantification of asymmetry of brain activity intensity during motor imagery tasks.

The resting state cortical functional connectivity is an important method in current brain researches. In this paper, we propose an approach for analyzing and manipulating the resting state functional magnetic resonance imaging (fMRI) data using spatial independent component analysis (sICA) method, and applying the low-frequency oscillations theory to the choice of component of interest (COI) from the component obtained by sICA method. Firstly, we remove all the inactive voxels and independent voxels via Z value. Then, by making a spectrum analysis, we choose the COI with concentrations of energy between 0.01 and 0.1 Hz. And after that, we obtain the functional connectivity networks using hierarchical clustering.

Brain activation during motor imagery has been studied extensively for years, but only a few of these studies focused on investigating the effective connectivity in the brain. The existence of interactions or closed loop circuits between the SMA and other brain regions during motor imagery still remains unclear. In the present study, selecting the SMA as the region of interest, we used the Granger causality mapping (GCM) method to explore the effective connectivity in the brain during motor imagery. Our results demonstrated that more brain regions showed effective connections to the SMA during the right-hand motor imagery than during the left-hand motor imagery, but the strength of the casual influence during the left-hand motor imagery was stronger than that of the right-hand motor imagery. We further found forward and backward effective connectivity between the SMA and three regions, including the bilateral dorsal premotor area (PMd), the contralateral primary and secondary somatosensory cortex (S1), and the primary motor cortex (M1). these results might indicate how the brain regions were inter-activated during motor imagery.

The self-organizing mapping (SOM) and hierarchical clustering (HC) methods are integrated to detect brain functional activation; functional magnetic resonance imaging (fMRI) data are first processed by SOM to obtain a primary merged neural nodes image, and then by HC to obtain further brain activation patterns. The conventional Euclidean distance metric was replaced by the correlation distance metric in SOM to improve clustering and merging of neural nodes. To improve the use of spatial and temporal information in fMRI data, a new spatial distance (node coordinates in the 2-D lattice) and temporal correlation (correlation degree of each time course in the exemplar matrix) are introduced in HC to merge the primary SOM results. Two simulation studies and two in vivo fMRI data that both contained block-design and event-related experiments revealed that brain functional activation can be effectively detected and that different response patterns can be distinguished using these methods. Our results demonstrate that the improved SOM and HC methods are clearly superior to the statistical parametric mapping (SPM), independent component analysis (ICA), and conventional SOM methods in the block-design, especially in the event-related experiment, as revealed by their performance measured by receiver operating characteristic (ROC) analysis. Our results also suggest that the proposed new integrated approach could be useful in detecting block-design and event-related fMRI data.