Model Interpretation Considering Both Time and Frequency Axes Given Time Series Data

Recently, deep learning-based models have emerged in the medical domain. Although those models achieve high performance, it is difficult to directly apply them in practice. Specifically, most models are not considered reliable yet, while they are not interpretable. Therefore, researchers attempt to interpret their own deep learning applications. However, the interpretation is task specific or only appropriate for image data such as computed tomography (CT) scans and magnetic resonance imaging (MRI). Currently, few works focus on the model interpretation given time series data such as electroencephalography (EEG) and electrocardiography (ECG) using LIME. Because the explanation generated by LIME is from the permutation of the divided input data, the performance of interpretation is highly dependent on the split method. In the medical domain, for the time series data, existing interpretations consider only the time axis, whereas physicians take account of the frequency too. In this work, we propose the model interpretation using LIME considering both time and frequency axes. Our key idea is that we divide the input signal using graph-based image clustering after transforming it using short-time Fourier transform, which is utilized to capture the change of frequency content over time. In our experiments, we utilize real-world data, which is EEG signals recorded from patients during polysomnographic (PSG) studies, as well as prove that ours captures a significantly more critical explanation than the state-of-the-art. In addition, we show that the representation obtained by ours reflects the physician's standard such as K-complexes and delta waves, which are considered strong evidence of the second sleep stage and a clue of the third sleep stage. We expect that our work can be applied to establish computer-aided diagnosis as well as to measure the reliability of deep learning models taking the time series into them.