Understanding complex physical systems often requires integrating data from multiple diagnostics, each with limited resolution or coverage. We present a machine learning framework that reconstructs synthetic high-temporal-resolution data for a target diagnostic using information from other diagnostics, without direct target measurements during the inference. This multimodal super-resolution technique improves diagnostic robustness and enables monitoring even in case of measurement failures or degradation. Applied to fusion plasmas, our method targets edge-localized modes (ELMs), which can damage plasma-facing materials. By reconstructing super-resolution Thomson Scattering data from complementary diagnostics, we uncover fine-scale plasma dynamics and validate the role of resonant magnetic perturbations (RMPs) in ELM suppression through magnetic island formation. The approach provides new observation supporting the plasma profile flattening due to these islands. Our results demonstrate the framework's ability to generate high-fidelity synthetic diagnostics, offering a powerful tool for ELM control development in future reactors like ITER. The approach is broadly transferable to other domains facing sparse, incomplete, or degraded diagnostic data, opening new avenues for discovery.

A non-linear complex system governed by multi-spatial and multi-temporal physics scales cannot be fully understood with a single diagnostic, as each provides only a partial view and much information is lost during data extraction. Combining multiple diagnostics also results in imperfect projections of the system's physics. By identifying hidden inter-correlations between diagnostics, we can leverage mutual support to fill in these gaps, but uncovering these inter-correlations analytically is too complex. We introduce a groundbreaking machine learning methodology to address this issue. Our multimodal approach generates super-resolution data encompassing multiple physics phenomena, capturing detailed structural evolution and responses to perturbations previously unobservable. This methodology addresses a critical problem in fusion plasmas: the Edge Localized Mode (ELM), a plasma instability that can severely damage reactor walls. One method to stabilize ELM is using resonant magnetic perturbation to trigger magnetic islands. However, low spatial and temporal resolution of measurements limits the analysis of these magnetic islands due to their small size, rapid dynamics, and complex interactions within the plasma. With super-resolution diagnostics, we can experimentally verify theoretical models of magnetic islands for the first time, providing unprecedented insights into their role in ELM stabilization. This advancement aids in developing effective ELM suppression strategies for future fusion reactors like ITER and has broader applications, potentially revolutionizing diagnostics in fields such as astronomy, astrophysics, and medical imaging.

Echo State Networks (ESNs) are a special type of Recurrent Neural Networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. However, recent publications have addressed the problem that a purely random initialization may not be ideal. Instead, a completely deterministic or data-driven initialized ESN structure was proposed. In this work, an unsupervised training methodology for the hidden components of an ESN is proposed. Motivated by traditional Hidden Markov Models (HMMs), which have been widely used for speech recognition for decades, we present an unsupervised pre-training method for the recurrent weights and bias weights of ESNs. This approach allows for using unlabeled data during the training procedure and shows superior results for continuous spoken phoneme recognition, as well as for a large variety of time-series classification datasets.

Echo state networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image, and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the $K$ -means algorithm on the training data. We show that for a large variety of datasets, this initialization performs equivalently or superior than a randomly initialized ESN while needing significantly less reservoir neurons. Furthermore, we discuss that this approach provides the opportunity to estimate a suitable size of the reservoir based on prior knowledge about the data.

The successful steady-state operation of burning fusion plasmas in planned future devices such as the ITER tokamak requires understanding of fast-ion physics. Alfven eigenmodes are special cases of plasma waves driven by fast ions that are important to identify and control since they can lead to loss of confinement and potential damage to the inner walls of a plasma device. The goal of this work is to compare machine learning-based systems trained to classify Alfven eigenmodes using CO2 interferometer data from a labelled database on the DIII-D tokamak. A Long-Short Term Memory (LSTM) network is trained from scratch using simple spectrogram representations of the CO2 phase data. The model is trained using a single chord (sequence) per training step. Results show a total true positive rate of = 90% and a false positive rate of = 18%. This paper demonstrates the potential of applying machine learning models to detect and identify different classes of Alfv ' en eigenmodes for real-time applications in steady-state plasma operations that could potentially drive actuators to mitigate Alfv ' en eigenmode impacts.

In recent years, Echo State Networks (ESNs), a special type of Recurrent Neural Networks (RNNs), have become increasingly established in the Machine Learning (ML) community due to their relatively simple initialization and training methods. Traditionally, the input and recurrent weights are generated randomly, with only the output weights being trained, typically using linear regression. However, recent publications have proposed alternative ways to initialize the weight matrices, e.g., by using more deterministic methods or data-driven approaches. This is the first work comparing different simple reservoir structures and an ESN with pre-trained input weights for the task of monitoring the state of a door using a surveillance camera in real-time. The results show that deterministic ESN structures perform better than the randomly initialized baseline, achieving a frame error rate of 2.62% vs. 2.93%.

A Machine-Learning (ML) based detection scheme that automatically detects Alfv & eacute;n Eigenmodes (AE) in a labelled DIII-D database is presented here. Controlling AEs is important for the success of planned burning plasma devices such as ITER, since resonant fast ions can drive AEs unstable and degrade the performance of the plasma or damage the first walls of the machine vessel. Artificial Intelligence could be useful for real-time detection and control of AEs in steady-state plasma scenarios by implementing ML-based models into control algorithms that drive actuators for mitigation of AE impacts. Thus, the objective is to compare differences in performance between using two different recurrent neural network systems (Reservoir Computing Network and Long Short Term Memory Network) and two different representations of the CO2 phase data (simple and crosspower spectrograms). All CO2 interferometer chords are used to train both models, but only one is processed during each training step. The results from the model and data comparison show higher performance for the RCN model (True Positive Rate = 90% and False Positive Rate = 14%), and that using simple magnitude spectrograms is sufficient to detect AEs. Also, the vertical CO2 interferometer chord passing near the center is better for ML-based detection of AEs.

Reservoir Computing Networks (RCNs) belong to a group of machine learning techniques that project the input space non-linearly into a high-dimensional feature space, where the underlying task can be solved linearly. Popular variants of RCNs are capable of solving complex tasks equivalently to widely used deep neural networks, but with a substantially simpler training paradigm based on linear regression. In this paper, we show how to uniformly describe RCNs with small and clearly defined building blocks, and we introduce the Python toolbox PyRCN (Python Reservoir Computing Networks) for optimizing, training and analyzing RCNs on arbitrarily large datasets. The tool is based on widely-used scientific packages and complies with the scikit-learn interface specification. It provides a platform for educational and exploratory analyses of RCNs, as well as a framework to apply RCNs on complex tasks including sequence processing. With a small number of building blocks, the framework allows the implementation of numerous different RCN architectures. We provide code examples on how to set up RCNs for time series prediction and for sequence classification tasks. PyRCN is around ten times faster than reference toolboxes on a benchmark task while requiring substantially less boilerplate code.

Automatic music transcription (AMT) is one of the challenging problems in Music Information Retrieval with the goal of generating a score-like representation of a polyphonic audio signal. Typically, the starting point of AMT is an acoustic model that computes note likelihoods from feature vectors. In this work, we evaluate the capabilities of Echo State Networks (ESNs) in acoustic modeling of piano music. Our experiments show that the ESN-based models outperform state-of-the-art Convolutional Neural Networks (CNNs) by an absolute improvement of 0.5 $$F_{1}$$ -score without using an extra language model. We also discuss that a two-layer ESN, which mimics a hybrid acoustic and language model, achieves better results than the best reference approach that combines Invertible Neural Networks (INNs) with a biGRU language model by an absolute improvement of 0.91 $$F_{1}$$ -score.

As a family member of Recurrent Neural Networks and similar to Long-Short-Term Memory cells, Echo State Networks (ESNs) are capable of solving temporal tasks, but with a substantially easier training paradigm based on linear regression. However, optimizing hyper-parameters and efficiently implementing the training process might be somewhat overwhelming for the first-time users of ESNs. This paper aims to facilitate the understanding of ESNs in theory and practice. Treating ESNs as non-linear filters, we explain the effect of the hyper-parameters using familiar concepts such as impulse responses. Furthermore, the paper introduces the Python toolbox PyRCN (Python Reservoir Computing Network) for developing, training and analyzing ESNs on arbitrarily large datasets. The tool is based on widely-used scientific packages, such as numpy and scipy and offers an interface to scikit-learn. Example code and results for classification and regression tasks are provided.

OBJECTIVE:Stroke survivors commonly suffer from dysphagia, originating from oro-facial impairments which affect swallowing function. Functional therapy often employs tongue exercises that require the patient to perform short motion sequences. Evaluating the patient's performance on those exercises is difficult, because there is no reliable form of visual feedback.METHODS:We propose an optopalatographic device that does not require a personalized dental retainer and is capable of measuring tongue movement trajectories intraorally. The device features nine optical proximity sensors at 100 Hz and is fixated against the hard palate with a specifically developed palatal adhesive. The sensing capabilities of the device were evaluated on a tongue gesture corpus recorded from nine healthy individuals, containing eight different tongue exercises commonly used in functional dysphagia therapy.RESULTS:The measured tongue trajectories contained temporally and spatially resolved information about the tongue movement and location during each exercise. Furthermore, a simple DTW-kNN classifier was able to distinguish the exercises from one another with an average classification accuracy of 97.9 % and 61.4 % (cross-validation and inter-speaker test accuracy, respectively).CONCLUSION:the device can provide real-time feedback for tongue motion and we obtained promising gesture recognition results with relatively few sensors, even in the absence of a personalized dental retainer.SIGNIFICANCE:Non-personalized optopalatography is readily available and could aid in improving functional dysphagia therapy by providing visual feedback to both the physician and patient.

The Time of Excitation (Tx) of speech, also widely known as the Glottal Closure Instants (GCI) denote the points in time at which the vocal folds close during the production of voiced speech. In this paper, we extend a previous approach based on a multilayer perceptron (MLP) using Echo State Networks (ESN), a variant of a Recurrent Neural Network (RNN). We show that the MLP and ESN approaches lead to similar results. The ESN model performed better than the MLP when the latter used only a single input sample (0.86 vs 0.75 area under the ROC plot), whereas the MLP slightly outperformed the ESN (0.98 vs 0.97 area under the ROC plot) when its was provided with a sufficient number of surrounding speech samples.

Note onset detection – the detection of the beginning of new note events – is a fundamental task for music analysis that can help to improve Automatic Music Transcription (AMT). The method for onset detection always follows a similar outline: An audio signal is transformed into an Onset Detection Function (ODF), which should have rather low values (i.e. close to zero) for most of the time, and pronounced peaks at onset times, which can then be extracted by applying peak picking algorithms on the ODF. Currently, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) define the state of the art. In this paper, we build upon previous work about onset detection using Echo State Networks (ESNs) that have achieved comparable results to CNNs. We show that unsupervised pre-training of the ESN leads to similar results whilst reducing the model complexity.

Currently, convolutional neural networks (CNNs) define the state of the art for multipitch tracking in music signals. Echo State Networks (ESNs), a recently introduced recurrent neural network architecture, achieved similar results as CNNs for various tasks, such as phoneme or digit recognition. However, they have not yet received much attention in the community of Music Information Retrieval. The core of ESNs is a group of unordered, randomly connected neurons, i.e., the reservoir, by which the low-dimensional input space is non-linearly transformed into a high-dimensional feature space. Because only the weights of the connections between the reservoir and the output are trained using linear regression, ESNs are easier to train than deep neural networks. This paper presents a first exploration of ESNs for the challenging task of multipitch tracking in music signals. The best results presented in this paper were achieved with a bidirectional two-layer ESN with 20 000 neurons in each layer. Although the final F-score of 0.7198 still falls below the state of the art (0.7370), the proposed ESN-based approach serves as a baseline for further investigations of ESNs in audio signal processing in the future.

Maximizing the utilization of grid components in operation is a major challenge for distribution system operators. In order to achieve this, assessment of the technical conditions of the equipment is required. The dielectric performance of insulation systems can be evaluated by the interpretation of Partial Discharge (PD) signals. Evaluation of the characteristics of these time-resolved signals is of highest importance. Conventional methods have several drawbacks, such as the complexity of the test setup and the settings as well as costs of the systems. Available PD measuring systems are not designed to face future challenges arising from Direct Current (DC)-infrastructure for energy distribution. To apply machine learning (ML) algorithms, typically the complete time signals have to be provided in order to extract features. In the present study, to allow a real-time processing at high signal resolutions, an automatically initializing algorithm was developed to separate noise and deterministic signals from each other. The PD or interference signals are available for extended signal processing such as clustering or classification. The implementation of this technology for Alternating Current (AC) and DC voltage applications was evaluated under practical conditions. Finally, an outlook on future applications and options using algorithms of ML in combination with the presented technology is given.

In music analysis, one of the most fundamental tasks is note onset detection - detecting the beginning of new note events. As the target function of onset detection is related to other tasks, such as beat tracking or tempo estimation, onset detection is the basis for such related tasks. Furthermore, it can help to improve Automatic Music Transcription (AMT). Typically, different approaches for onset detection follow a similar outline: An audio signal is transformed into an Onset Detection Function (ODF), which should have rather low values (i.e. close to zero) for most of the time but with pronounced peaks at onset times, which can then be extracted by applying peak picking algorithms on the ODF. In the recent years, several kinds of neural networks were used successfully to compute the ODF from feature vectors. Currently, Convolutional Neural Networks (CNNs) define the state of the art. In this paper, we build up on an alternative approach to obtain a ODF by Echo State Networks (ESNs), which have achieved comparable results to CNNs in several tasks, such as speech and image recognition. In contrast to the typical iterative training procedures of deep learning architectures, such as CNNs or networks consisting of Long-Short-Term Memory Cells (LSTMs), in ESNs only a very small part of the weights is easily trained in one shot using linear regression. By comparing the performance of several feature extraction methods, pre-processing steps and introducing a new way to stack ESNs, we expand our previous approach to achieve results that fall between a bidirectional LSTM network and a CNN with relative improvements of 1.8 % and -1.4 %, respectively. For the evaluation, we used exactly the same 8-fold cross validation setup as for the reference results.

Early detection of malign patterns in patients’ biological signals can save millions of lives. Despite the steady improvement of artificial intelligence–based techniques, the practical clinical application of these methods is mostly constrained to an offline evaluation of the patients’ data. Previous studies have identified organic electrochemical devices as ideal candidates for biosignal monitoring. However, their use for pattern recognition in real time was never demonstrated. Here, we produce and characterize brain-inspired networks composed of organic electrochemical transistors and use them for time-series predictions and classification tasks using the reservoir computing approach. To show their potential use for biofluid monitoring and biosignal analysis, we classify four classes of arrhythmic heartbeats with an accuracy of 88%. The results of this study introduce a previously unexplored paradigm for biocompatible computational platforms and may enable development of ultralow–power consumption hardware-based artificial neural networks capable of interacting with body fluids and biological tissues.

In music analysis, one of the most fundamental tasks is note onset detection – detecting the beginning of new note events. It is the basis for more high-level tasks, such as beat tracking or tempo detection. The main outline of all approaches for onset detection is roughly the same: The audio signal is transformed into an Onset Detection Function (ODF), which is zero for most of the time but has pronounced peaks in case of onsets. Applying peak picking algorithms on the ODF, the onset times can be extracted. Currently, Convolutional Neural Networks (CNNs) define the state of the art. In this paper, a first exploration of Echo State Networks (ESNs) to obtain an ODF is presented. ESNs have achieved comparable results to CNNs in several recognition tasks, such as speech and image recognition. Features were extracted using a bank of filters with a logarithmic frequency spacing. The feature vectors were fed into the ESN that computed the ODF. Applying a simple threshold-based peak picking algorithm on the ODF, the onsets were detected. For the hyperparameter optimization, a dataset with pre-defined splits for an 8-fold cross validation was used. With all hyperparameters optimized, we reached an F-Measure of 0.812 using a bidirectional ESN with 8000 neurons.

In this paper we propose a new analysis-by-synthesis algorithm for multipitch tracking. It uses spectral base components extracted from isolated piano notes to model a spectrum. The advantage compared to other signal-based algorithms is the application of musical knowledge for the identification process. The PianoTranscriber uses a limited set of the 88 musical notes of the standard piano. The extracted time series for all possible notes are used for a resynthesis step. Original and resynthesized signals are compared using an adapted onset-detection function. The PianoTranscriber was parametrized, optimized and evaluated using subsets of the publicly available MAPS-database. For the evaluation, the PianoTranscriber was compared to the state-of-the-art algorithm SONIC. The results show that the PianoTranscriber outperformed SONIC using base components from the analyzed piano and achieved similar results using base components from other pianos.