Deep reinforcement learning (DRL) has shown remarkable capabilities in solving sequential decision-making problems. However, DRL requires extensive interactions with image-based environments. Existing methods have combined self-supervised learning or data augmentation to improve sample efficiency. While understanding the temporal information dynamics of the environment is important for effective learning, many methods do not consider these factors. To address the sample efficiency problem, we propose dynamics modeling with sequential temporal information (DynaSTI) that incorporates environmental dynamics and leverages the correlation among trajectories to improve sample efficiency. DynaSTI uses an effective learning strategy for state representation as an auxiliary task, using gated recurrent units to capture temporal information. It also integrates forward and inverse dynamics modeling in a hierarchical configuration, enhancing the learning of environmental dynamics compared to using each model separately. The hierarchical structure of DynaSTI enhances the stability of inverse dynamics modeling during training by using inputs derived from forward dynamics modeling, which focuses on feature extraction related to controllable state. This approach effectively filters out noisy information. Consequently, using denoised inputs from forward dynamics modeling results in improved stability when training inverse dynamics modeling, rather than using inputs directly from the encoder. We demonstrate the effectiveness of DynaSTI through experiments on the Atari game benchmark, limiting the environment interactions to 100k steps. Our extensive experiments confirm that DynaSTI significantly improves the sample efficiency of DRL, outperforming comparison methods in terms of statistically reliable metrics and nearing human-level performance.

In pixel-based deep reinforcement learning (DRL), learning representations of states that change because of an agent’s action or interaction with the environment poses a critical challenge in improving data efficiency. Recent data-efficient DRL studies have integrated DRL with self-supervised learning (SSL) and data augmentation to learn state representations from given interactions. However, some methods have difficulties in explicitly capturing evolving state representations or in selecting data augmentations for appropriate reward signals. Our goal is to explicitly learn the inherent dynamics that change with an agent’s intervention and interaction with the environment. We propose masked and inverse dynamics modeling (MIND), which uses masking augmentation and fewer hyperparameters to learn agent-controllable representations in changing states. Our method is comprised of a self-supervised multitask learning that leverages a transformer architecture, which captures the spatiotemporal information underlying in the highly correlated consecutive frames. MIND uses two tasks to perform self-supervised multitask learning: masked modeling and inverse dynamics modeling. Masked modeling learns the static visual representation required for control in the state, and inverse dynamics modeling learns the rapidly evolving state representation with agent intervention. By integrating inverse dynamics modeling as a complementary component to masked modeling, our method effectively learns evolving state representations. We evaluate our method by using discrete and continuous control environments with limited interactions. MIND outperforms previous methods across benchmarks and significantly improves data efficiency. The code is available at https://github.com/dudwojae/MIND.

With the development of deep learning technology, deep reinforcement learning (DRL) has successfully built intelligent agents in sequential decision-making problems through interaction with image-based environments. However, learning from unlimited interaction is impractical and sample inefficient because training an agent requires many trial and error and numerous samples. One response to this problem is sample-efficient DRL, a research area that encourages learning effective state representations in limited interactions with image-based environments. Previous methods could effectively surpass human performance by training an RL agent using self-supervised learning and data augmentation to learn good state representations from a given interaction. However, most of the existing methods only consider similarity of image observations so that they are hard to capture semantic representations. To address these challenges, we propose spatio-temporal and action-based contrastive representation (STACoRe) learning for sample-efficient DRL. STACoRe performs two contrastive learning to learn proper state representations. One uses the agent's actions as pseudo labels, and the other uses spatio-temporal information. In particular, when performing the action-based contrastive learning, we propose a method that automatically selects data augmentation techniques suitable for each environment for stable model training. We train the model by simultaneously optimizing an action-based contrastive loss function and spatio-temporal contrastive loss functions in an end-to-end manner. This leads to improving sample efficiency for DRL. We use 26 benchmark games in Atari 2600 whose environment interaction is limited to only 100k steps. The experimental results confirm that our method is more sample efficient than existing methods. The code is available at https://github.com/dudwojae/STACoRe.

Detecting an anomaly in multichannel signal data is a challenging task in various domains. It should take into account the cross-channel relationship and temporal relationship within each channel. Moreover, the signal data is high-dimensional and making it difficult to gather sufficient abnormal labels. Consequently, unsupervised reconstruction-based anomaly detection methods have been applied successfully in many studies. However, they lose valuable channel information inherent in the reconstruction errors by merely averaging the errors for both the channel and time, then consider the average value as an anomaly score. In this study, we propose a method to explicitly employ channelwise reconstruction errors as a feature to detect abnormal signals. After a convolutional autoencoder produces the channelwise reconstruction errors, a machine learning anomaly detection model aggregates the errors as an anomaly score. To demonstrate the effectiveness and applicability of the proposed model, we conduct experiments using simulated data and real-world automobile data. The results show that the proposed method remarkably enhances the detectability compared to the simple average of the reconstruction errors. The reconstruction errors of abnormal and normal channels are shown to be different; therefore, it can be considered as an appropriate feature for anomaly detection. The best performance is obtained by using local outlier factors in the following anomaly detection model.

Multichannel signal data collected from multiple sensors are widely used to monitor the status of various mechanical systems. Recently, deep neural networks have been success-fully applied to multichannel signal data analysis because of their capability to learn dis-criminative features with minimum feature engineering. However, the latest deep neural networks for multichannel signal analysis lack explainability, which is essential for post hoc analysis in various fields. In this study, we propose an explainable neural network for the multichannel signal classification task. The proposed method is equipped with two levels of attention mechanisms -at the segment and channel levels- encouraging the model to focus on important parts in discriminating the status of a system. The derived attention probabilities facilitate interpretation of network behavior and thus can support post hoc analysis. To demonstrate the practicality and applicability of the proposed method, we conducted experiments on both simulated and real-world automobile data. The results confirmed that the proposed method is capable of accurately classifying mul-tichannel signals and correctly identifying the critical segments and channels. (c) 2021 Elsevier Inc. All rights reserved. With recent advances in sensor technology regarding hardware and software for data storage and wireless communication, the use of multiple sensors of monitoring of various systems based on multiple sensors has become more prevalent than ever. Signals collected from the sensors can be used to describe states or to detect system malfunctions of a system. Examples include human activity recognition [1], automobile statement recognition [2], and monitoring construction equipment monitoring [3]. Various methods have been considered for analyzing complex multivariate time series data obtained from multiple sen

Anomaly detection is the task of learning patterns of normal data and identifying data with other characteristics. As various types of sensors are attached to vehicle, healthcare equipment, production facilities, etc., detecting anomalies in multi-channel sensor data has become very important. In sensor data, abnormal signals occur temporally during certain intervals of a few channels. It is very important to capture the characteristics of individual channel and cross-channel relationship in order to detect abnormal signals that occur locally for a short time interval. We propose a channel-wise reconstruction-based anomaly detection framework which consists of two parts: channel-wise reconstruction part with convolutional autoencoder (CAE) and anomaly scoring part with machine learning algorithms, isolation forest (iForest) and local outlier factor (LOF). CAE learns the features of normal signal data and measures channel-wise reconstruction error. We applied the symmetric skip-connections technique to build a CAE model for higher reconstruction performance. Given the channel-wise reconstruction error as an input, iForest and LOF summarize it to anomaly score. We present our results on data collected from real sensors attached to vehicle and show that the proposed framework outperforms traditional reconstruction-based anomaly detection methods and one-class classification methods.

In many industries, statistical process monitoring techniques play a key role in improving processes through variation reduction and defect prevention. Modern large-scale industrial processes require appropriate monitoring techniques that can efficiently address high-dimensional nonlinear processes. Such processes have been successfully monitored with several latent variable-based methods. However, because these monitoring methods use Hotelling’s T2 statistics in the reduced space, a normality assumption underlies the construction of these tools. This assumption has limited the use of latent variable-based monitoring charts in both nonlinear and nonnormal situations. In this study, we propose a variational autoencoder (VAE) as a monitoring method that can address both nonlinear and nonnormal situations in high-dimensional processes. VAE is appropriate for T2 charts because it causes the reduced space to follow a multivariate normal distribution. The effectiveness and applicability of the proposed VAE-based chart were demonstrated through experiments on simulated data and real data from a thin-film-transistor liquid-crystal display process.