Learning computer programming is a demanding cognitive task that requires a number of competencies. Students with limited problem-solving skills or prior programming knowledge find it challenging to understand the abstract concepts involved. While much research has investigated how cognitive load affects learning, few studies have investigated the cognitive load in memory associated with programming tasks using electroencephalograms. This research uses the EEG data, modelled in the NeuCube spiking neural network architecture, to analyse the memory efficiency in two groups of students, with and without prior programming knowledge, when learning a new language. Based on the quantitative analysis of brain neuronal connectivity captured in the NeuCube models, it is concluded that prior programming knowledge results in less cognitive load, meaning more efficient memory use. Findings and results of empirical data analysis in this study can inform educators to develop strategies and design personalised learning interventions based on the student’s prior knowledge.

Current technological solutions for Brain-machine Interfaces (BMI) achieve reasonable accuracy, but most systems are large in size, power consuming and not auto-adaptive. This work addresses the question whether current neuromorphic technologies could resolve these problems? The paper proposes a novel neuromorphic framework of a BMI system for prosthetics control via decoding Electro Cortico-Graphic (ECoG) brain signals. It includes a three-dimensional spike timing neural network (3D-SNN) for brain signals features extraction and an on-line trainable recurrent reservoir structure (Echo state network (ESN)) for Motor Control Decoding (MCD). A software system, written in Python using NEST Simulator SNN library is described. It is able to adapt continuously in real time in supervised or unsupervised mode. The proposed approach was tested on several experimental data sets acquired from a tetraplegic person. First simulation results are encouraging, showing also the need for a further improvement via multiple hyper-parameters tuning. Its future implementation on a neuromorphic hardware platform that is smaller in size and significantly less power consuming is discussed too.

In our previous work, we developed a neuromorphic decoder of intended movements of tetraplegic patients using ECoG recordings from the brain motor cortex, called Motor Control Decoder (MCD). Even though the training data are labeled based on the desired movement, there is no guarantee that the patient is satisfied by the action of the effectors. Hence, the need for the classification of brain signals as satisfactory/unsatisfactory is obvious. Based on previous work, we upgrade our neuromorphic MCD with a Neural Response Decoder (NRD) that is intended to predict whether ECoG data are satisfactory or not in order to improve MCD accuracy. The main aim is to design an actor–critic structure able to adapt via reinforcement learning the MCD (actor) based on NRD (critic) predictions. For this aim, NRD was trained using not only an ECoG signal but also the MCD prediction or prescribed intended movement of the patient. The achieved accuracy of the trained NRD is satisfactory and contributes to improved MCD performance. However, further work has to be carried out to fully utilize the NRD for MCD performance optimization in an on-line manner. Possibility to include feedback from the patient would allow for further improvement of MCD-NRD accuracy.

Quantum machine learning (QML) has gained significant attention recently for exploring the quantum computing applications. In this work, we delve into QML applications using the quantum support vector classifier (QSVC). QSVC uses quantum feature maps to create quantum-enhanced kernels. These kernels have the potential to provide a quantum advantage over classical machine learning (ML) algorithms in data classification. In this study, we introduced a novel quantum feature map and analyzed its role in developing a quantum-enhanced classifier, thereby expanding the applications of QML. Additionally, the study explored the importance of hyperparameter tuning for achieving quantum enhancement. To demonstrate proof of concept, we analyzed motor imagery electroencephalogram (EEG) datasets using the quantum kernel. We compared the results from our novel quantum feature map with outcomes from state-of-the-art feature maps. Additionally, we assessed the performance of quantum kernels in comparison with classical kernels and other ML classifiers. Finally, the study demonstrates the analytical advantages of the proposed quantum feature map over state-of-the-art feature maps and various classical classifiers through classification accuracy.

The Epileptor model is a mathematical framework utilized for simulating the transition from interictal to ictal local field potential (LFP) activity in the brain, with the aim of predicting and preventing epileptic seizures. This article introduces a novel approach integrating Lyapunov and Poincaré–Bendixson methods to analyze the stability of limit cycles in nonlinear systems, specifically focusing on Epileptors with a two-state dynamic. Our method accurately delineates the limit cycle boundary through eigenvalue-based analysis, facilitating precise assessment of stability properties and identification of critical regions linked to seizure initiation and termination. Through the investigation of the two-state dynamics of Epileptors, we gain deeper insights into the transition between low activity and seizure states, consequently improving our understanding of epileptic seizures. Our approach can be employed to establish stability conditions and determine the existence of limit cycles in Epileptor models, which can further aid in predicting and preventing epileptic seizures by identifying critical regions associated with seizure initiation and termination. The simulations conducted in this study demonstrate that the model under investigation exhibits stable limit cycle behavior and manifests bifurcation, with significant implications for the development of targeted interventions and more effective prediction and treatments for epilepsy. The findings indicate that the suggested approach establishes that external stimulation should not surpass 10.8 mA. Moreover, the initial normal state lies within the range of $-$ 1.6 to $-$ 0.1 ictal LFP. On the other hand, the LaSalle and eigenvalue methods individually cannot precisely determine the limit cycle region.

Personalised modelling has become dominant in personalised medicine and precision health. It creates a computational model for an individual based on large data repositories of existing personalised data, aiming to achieve the best possible personal diagnosis or prognosis and derive an informative explanation for it. Current methods are still working on a single data modality or treating all modalities with the same method. The proposed method, SAIN (Search-And-INfer), offers better results and an informative explanation for classification and prediction tasks on a new multimodal object (sample) using a database of similar multimodal objects. The method is based on different distance measures suitable for each data modality and introduces a new formula to aggregate all modalities into a single vector distance measure to find the closest objects to a new one, and then use them for a probabilistic inference. This paper describes SAIN and applies it to two types of multimodal data, cardiovascular diagnosis and EEG time series, modelled by integrating modalities, such as numbers, categories, images, and time series, and using a software implementation of SAIN.

The process of fusing two complementary data, panchromatic and multispectral images, to create high-resolution multispectral (HRMS) images is known as pansharpening. Combining detail injection (DI) methods with convolutional neural networks (CNN) for improved HRMS image fusion quality is a research hotspot due to their interpretability and large-scale data processing capabilities, respectively. Nevertheless, the current hybrid models typically concatenate CNN and traditional techniques, limiting the ability to utilize the benefits of both approaches. This paper presents a new hybrid network, multispectral pansharpening method based on CNN-DI network with mixture of experts (CDN-MoE), using detail injection theory to design a deep learning framework. Specifically, we first create the mixture of detail inject experts network (MoDIE-Net) that mixes training pairs of full-and reduced-resolution images to enhance model generalization. Next, the adaptive correlation residual network (ACR-Net) is suggested to find the correlation between the spectral and spatial features of the source images. Finally, the global information injection network (GII-Net) is established to strengthen the accuracy of fusion results by integrating the context of input images. Additionally, to reduce the loss of spectral features during the upsampling process, the spectral reconstruction network (SR-Net) is proposed. We perform both qualitative and quantitative experiments on the GaoFen-2, IKONOS, and WorldView-2 datasets at various resolutions. Our approach has advantages over other SOTA pansharpening methods currently available in terms of visual effects and objective metrics.

This paper introduces a novel framework, called here 'NeuDen' for the integration of neuromorphic evolving spiking neural networks (eSNN), that learn efficiently multiple time series in their temporal association and interaction as spike based information, with dynamic evolving neuro-fuzzy systems (deNFS), that learn incrementally extracted from the eSNN frequency-based (rate-based) feature vectors, to predict future time-series values and to produce interpretable fuzzy rules. The new framework aims to make the best out of the dominant characteristics of the two types of models. First, spike-time-dependent plasticity (STDP) learning is used in SNN to learn temporal interaction between multiple time series, connected to a dynamic eSNN (deSNN) as a regressor/classifier. Then, frequency-based feature-vectors are extracted from the trained deSNN for further learning, fuzzy inference and rule extraction in a deNFS, here exemplified by a popular DENFIS model, resulting in an accurate prediction results and explainable dynamic fuzzy rules. The NeuDen, framework and model, overcomes both the explainability problems of eSNN and the limitations of deNFS to model multiple streaming time series in their temporal interaction. NeuDen surpasses both deSNN and DENFIS by providing multiple regression models and achieving higher accuracy. NeuDen is demonstrated on benchmark data and on financial and economic time series, achieving from 3 to 100 times smaller RMSE when compared with other evolving systems. The proposed framework opens a new direction for the development of more efficient evolving systems by integrating eSNN with other Explainable Artificial Intelligence (XAI) techniques, such as other neuro-fuzzy systems, deep neural networks, and quantum classifiers for specific applications.

At present, multimodal medical image fusion technology has become an essential means for researchers and doctors to predict diseases and study pathology. Nevertheless, how to reserve more unique features from different modal source images on the premise of ensuring time efficiency is a tricky problem. To handle this issue, we propose a flexible semantic-guided architecture with a mask-optimized framework in an end-to-end manner, termed as GeSeNet. Specifically, a region mask module is devised to deepen the learning of important information while pruning redundant computation for reducing the runtime. An edge enhancement module and a global refinement module are presented to modify the extracted features for boosting the edge textures and adjusting overall visual performance. In addition, we introduce a semantic module that is cascaded with the proposed fusion network to deliver semantic information into our generated results. Sufficient qualitative and quantitative comparative experiments (i.e., MRI-CT, MRI-PET, and MRI-SPECT) are deployed between our proposed method and ten state-of-the-art methods, which shows our generated images lead the way. Moreover, we also conduct operational efficiency comparisons and ablation experiments to prove that our proposed method can perform excellently in the field of multimodal medical image fusion. The code is available at https://github.com/lok-18/GeSeNet.

The study introduces a new online spike encoding algorithm for spiking neural networks (SNN) and suggests new methods for learning and identifying diagnostic biomarkers using three prominent deep learning neural network models: deep BiLSTM, reservoir SNN, and NeuCube. EEG data from datasets related to epilepsy, migraine, and healthy subjects are employed. Results reveal that BiLSTM hidden neurons capture biological significance, while reservoir SNN activities and NeuCube spiking dynamics identify EEG channels as diagnostic biomarkers. BiLSTM and reservoir SNN achieve 90 and 85% classification accuracy, while NeuCube achieves 97%, all methods pinpointing potential biomarkers like T6, F7, C4, and F8. The research bears implications for refining online EEG classification, analysis, and early brain state diagnosis, enhancing AI models with interpretability and discovery. The proposed techniques hold promise for streamlined brain-computer interfaces and clinical applications, representing a significant advancement in pattern discovery across the three most popular neural network methods for addressing a crucial problem. Further research is planned to study how early can these diagnostic biomarkers predict an onset of brain states.

Stroke is a medical condition that affects around 15 million people annually. Patients and their families can face severe financial and emotional challenges as it can cause motor, speech, cognitive, and emotional impairments. Stroke lesion segmentation identifies the stroke lesion visually while providing useful anatomical information. Though different computer-aided software are available for manual segmentation, state-of-the-art deep learning makes the job much easier. This review paper explores the different deep-learning-based lesion segmentation models and the impact of different pre-processing techniques on their performance. It aims to provide a comprehensive overview of the state-of-the-art models and aims to guide future research and contribute to the development of more robust and effective stroke lesion segmentation models.

Heavy snow seriously degrades the performance of outdoor computer vision systems. Near- and far-field snowflakes in heavy snow videos exhibit distinctly disparate physical properties. To address this issue, this research proposes a video desnowing model that utilizes stable tensor decomposition with snow depth prior information. Initially, the depths of snowflakes are transformed by their speeds calculated from the optical flow field of snowfall. Next, the noise level in backgrounds is available by dense snow. Finally, inspired by biomimicry, the snow on moving objects (MOs) is removed by the adaptive mimesis region of interest (AM-ROI). In contrast to previous tensor decomposition (TD), the introduction of a noise term enhances the stability of depth prior-based stable tensor decomposition (DP-STD) in addressing heavy snowfall. This approach enables more efficient and accurate restoration of the underlying nonsnow structure. Both synthetic and real snowfall experimental results show that our proposed desnowing model is more effective than the current SOTA algorithm in removing heavy snow.

Distinguishing stable and fluctuating psychopathological features in young individuals at Ultra High Risk (UHR) for psychosis is challenging, but critical for building robust, accurate, early clinical detection and prevention capabilities. Over a 24-month period, 159 UHR individuals were assessed using the Positive and Negative Symptom Scale (PANSS). Generalisability Theory was used to validate the PANSS with this population and to investigate stable and fluctuating features, by estimating the reliability and generalisability of three factor (Positive, Negative, and General) and five factor (Positive, Negative, Cognitive, Depression, and Hostility) symptom models. Acceptable reliability and generalisability of scores across occasions and sample population were demonstrated by the total PANSS scale (Gr = 0.85). Fluctuating symptoms (delusions, hallucinatory behaviour, lack of spontaneity, flow in conversation, emotional withdrawal, and somatic concern) showed high variability over time, with 50-68% of the variance explained by individual transient states. In contrast, more stable symptoms included excitement, poor rapport, anxiety, guilt feeling, uncooperativeness, and poor impulse control. The 3-factor model of PANSS and its subscales showed robust reliability and generalisability of their assessment scores across the UHR population and evaluation periods (G = 0.77-0.93), offering a suitable means to assess psychosis risk. Certain subscales within the 5-factor PANSS model showed comparatively lower reliability and generalisability (G = 0.33-0.66). The identified and investigated fluctuating symptoms in UHR individuals are more amendable by means of intervention, which could have significant implications for preventing and addressing psychosis. Prioritising the treatment of fluctuating symptoms could enhance intervention efficacy, offering a sharper focus in clinical trials. At the same time, using more reliable total scale and 3 subscales can contribute to more accurate assessment of enduring psychosis patterns in clinical and experimental settings.

This work is demonstrating the use of a supercomputer platform to optimise hyper-parameters of a proposed by the team novel SNN-ESN computational model, that combines a brain template of spiking neurons in a spiking neural network (SNN) for feature extraction and an Echo State Network (ESN) for dynamic data series classification. A case study problem and data are used to illustrate the functionalities of the SNN-ESN. The overall SNN-ESN classifier has several hyper-parameters that are subject to refinement, such as: spiking threshold, duration of the refractory period and STDP learning rate for the SNN part; reservoir size, spectral radius of the connectivity matrix and leaking rate for the ESN part. In order to find the optimal hyper-parameter values exhaustive search over all possible combinations within reasonable intervals was performed using supercomputer Avitohol. The resulted optimal parameters led to improved classification accuracy. This work demonstrates the importance of model parameter optimisation using a supercomputer platform, which improves the usability of the proposed SNN-ESN for real-time applications on complex spatio-temporal data.

The serious concerns over the negative impacts of Deepfakes have attracted wide attentions in the community of multimedia forensics. The existing detection works achieve deepfake detection by improving the traditional backbone networks to capture subtle manipulation traces. However, there is no attempt to construct new backbone networks with different structures for Deepfake detection by improving the internal feature representation of convolution. In this work, we propose a novel Space-Frequency Interactive Convolution (SFIConv) to efficiently model the manipulation clues left by Deepfake. To obtain high-frequency features from tampering traces, a Multichannel Constrained Separable Convolution (MCSConv) is designed as the component of the proposed SFIConv, which learns space-frequency features via three stages, namely generation, interaction and fusion. In addition, SFIConv can replace the vanilla convolution in any backbone networks without changing the network structure. Extensive experimental results show that seamlessly equipping SFIConv into the backbone network greatly improves the accuracy for Deepfake detection. In addition, the space-frequency interaction mechanism does benefit to capturing common artifact features, thus achieving better results in cross-dataset evaluation. Our code will be available at https://github.com/EricGzq/SFIConv .

Infrared and visible image fusion is critical in computer vision applications to integrate complementary information from input images into a single enhanced representation. However, existing infrared and visible image fusion approaches frequently overlook the joint requirements of feature transmission and advanced semantics. Consequently, they suffer from low feature utilization and weak generalization ability. Additionally, these approaches tend to concentrate on capturing global or local features while not understanding features from the perspective of the frequency components. This limitation makes it challenging to capture local frequency information accurately. To tackle these challenges, a novel joint training framework called Distillation-Fusion-Segmentation (DFSFuse) addresses the challenges of feature transfer, integration, and semantic comprehension in image fusion. This framework integrates the capabilities of knowledge transfer and semantic reasoning. The teacher and student networks ensure feature acquisition and processing during distillation. Specifically, the teacher network comprehends features from both a global perspective and frequency components and transfers these features to the student network to facilitate and oversee the learning process. Student networks specialize in extracting and reconstructing features from various modalities while efficiently learning and utilizing intermediate and fused features. Segmentation networks are employed to identify semantic content, emphasizing regions with rich semantics. Furthermore, we introduce a distilled semantic loss function to facilitate feature transfer efficiently. The experimental results indicate that our approach yields promising outcomes, achieving plausible visual effects compared to existing methods.

Sand-dust videos obtained in a low-light environment are characterized by low contrast, nonuniform illumination, color cast, and considerable noise. To realize sand-dust removal and brightness enhancement simultaneously, this article proposes an online low-light sand-dust video enhancement method using adaptive dynamic brightness correction and a rolling guidance filter. The proposed dual-threshold interframe detection strategy involves two methods to treat low-light sand-dust video frames. The first method involves two components: an adaptive dynamic brightness correction algorithm to correct the color deviation of the low-light video frame and improve its brightness and a rolling guidance filter combined with guided image filtering to enhance the frame details. The second method enhances the quality of the incoming frame by reducing the amount of calculation. The first frame of the video is processed using the first method. The processing method of each subsequent frame is determined according to its interframe detection value with the buffer frame. Through qualitative and quantitative comprehensive experiments on low-light sand-dust images and videos, the performance of the proposed method is compared with those of state-of-the-art methods. The proposed method for frame quality improvement achieves the best visual effect in enhancing the quality of low-light sand-dust images, as indicated by the best objective evaluation indicators. Moreover, compared with the framewise enhancement method, the video processing efficiency associated with the dual-threshold interframe detection strategy is 2.77 times higher.

Associative memories (AM) are at the core of human intelligence and learning systems. While there have been some neural network AM developed for vector-based data such as images, current machine learning methods, including deep neural networks, do not allow for training a model on time series data and recalling it on a subset of variables measured over a shorter time window. They also do not support further incremental training of the model on new temporal data and new variables. This paper introduces a new framework and method for the creation of evolving predictive associative memories for time series, abbreviated here as ePAMeT. The method is based on spiking neural networks (SNN). ePAMeT introduces significant adaptability in handling time series data with reduced or newly introduced features. This model maintains high accuracy and explainability, offering substantial improvements over traditional methods in dynamic and uncertain environments. First, an SNN model is trained on multiple time series using all available variables measured at a full-time length, and then the model is recalled on subsets of variables at a shorter time measurement without compromising predictive accuracy. Using a shorter time for recall makes early prediction of events possible. The SNN model can be further adapted/evolved on new data without pre-training the model on the old data, even using new variables. This is possible due to the evolving connectivity of the SNN model. A dynamic graph is extracted from the SNN model to capture dynamic interactions between the used temporal variables at any time during the evolution of the model, which constitutes strong explainability and a generation of new knowledge. The method is illustrated on original financial time series data, but it is applicable to many other domain areas as discussed. The proposed method has advantages over traditional machine learning methods in terms of evolvability, explainability, knowledge discovery, and using partial information of both the number of variables and their time length for the recall of the model on new data. The proposed framework opens the field for creating new types of evolvable time series prediction models. Future developments are discussed.

High-resolution multispectral (HRMS) images combine spatial and spectral information originating from panchromatic (PAN) and reduced-resolution multispectral (LRMS) images. Pansharpening performs well and is widely used to obtain HRMS images. However, most pansharpening approaches determine the ratio of PAN and LRMS images through direct interpolation, which may introduce artifacts and distort the color of the fused results. To address this issue, an unsupervised progressive pansharpening framework, MSBANet, is proposed, which adopts a multi-stage fusion strategy. Each stage contains an attention interactive extraction module (AIEM) and a multiscale bilateral fusion module (MBFM). The AIEM extracts spatial and spectral features from input images and captures the correlations between features. The MBFM can efficiently integrate information from the AIEM and improve MSBANet context awareness. We design a hybrid loss function that enhances the ability of the fusion network to store spectral and texture details. In qualitative and quantitative experimental studies on four datasets, MSBANet outperformed state-of-the-art pansharpening techniques. The code will be released.

Abstract This paper introduces a novel framework, called here 'NeuDen' for the integration of neuromorphic evolving spiking neural networks (eSNN), that learn efficiently multiple time series in their temporal association and interaction, with dynamic evolving neuro-fuzzy systems (deNFS), that learn incrementally extracted from the eSNN feature vectors, to predict future time-series values and to produce interpretable fuzzy rules. The new framework aims to make the best out of the dominant features of the two types of models. First, spike-time-dependent plasticity (STDP) learning is used in SNN to learn temporal interaction between multiple time series, connected to a dynamic eSNN (deSNN) as a regressor/classifier. Then, feature-vectors are extracted from the trained deSNN for further learning, fuzzy inference and rule extraction in a deNFS, here exemplified by DENFIS, resulting in an accurate prediction results and explainable dynamic fuzzy rules. The NeuDen, framework and model, overcomes both the explainability problems of eSNN and the limitations of deNFS to model multiple streaming time series in their temporal interaction. NeuDen surpasses both deSNN and DENFIS by providing multiple regression models and achieving higher accuracy. NeuDen is demonstrated on bench mark data and on financial and economic time series, achieving from 3 to 100 times smaller RMSE when compared with other evolving systems. The proposed framework opens a new direction for the development of more efficient evolving systems by integrating eSNN with other methods, such as other neuro-fuzzy systems, deep neural networks and quantum classifiers for specific applications.