Although most previous studies in cognitive neuroscience have focused on the change of the neuronal firing rate under various conditions, there has been increasing evidence that indicates the importance of neuronal oscillatory activities in cognition. In the visual cortex, specific time-frequency bands are thought to have selectivity to visual stimuli. Furthermore, several recent studies have shown that several time-frequency bands are related to frequency-specific feedforward or feedback processing in inter-areal communication. However, few studies have investigated detailed visual selectivity of each time-frequency band, especially in the primate inferior temporal cortex (ITC). In this work, we analyze frequency-specific electrocorticography (ECoG) activities in the primate ITC by training encoding models that predict frequency-specific amplitude from hierarchical visual features extracted from a deep convolutional neural network (CNNs). We find that ECoG activities in two specific time-frequency bands, theta (around 5 Hz) and gamma (around 20-25 Hz) bands, are better predicted from CNN features than the other bands. Furthermore, theta- and gamma-band activities are better predicted from higher and lower layers in CNNs, respectively. Our visualization analysis using CNN-based encoding models qualitatively show that theta- and gamma-band encoding models have selectivity to higher- and lower-level visual features, respectively. Our results suggest that neuronal oscillatory activities in theta and gamma bands carry distinct information in the hierarchy of visual features, and that distinct levels of visual information are multiplexed in frequency-specific brain signals.

Accurate decoding of perceptual information from brain signals is crucial in real-world BCI applications. While existing decoding methods work well in static, single-subject cases, more versatile, multi-subject decoding methods should be developed for achieving scalable and transferable BCI systems. In practice, it is not straightforward to record brain signals using the same recording equipment from a large number of subjects. If a pretrained decoder is not robust to subject or channel shifts, it cannot be applied to data from novel subjects and even from trained subjects when the recording equipment changes. In this work, we study brain decoding across multiple subjects with a different number of recording channels and channel location shifts. We consider channel-agnostic brain decoding as a multi-instance learning problem, where each input is seen as a set of instances. We propose a novel decoder architecture based on three building blocks: a channel-wise transform, an across-channel transform, and multi-channel pooling. We conduct a thorough experiment on our multi-subject electrocorticography (ECoG) classification dataset to verify the effectiveness of our proposed methods against other baseline architectures. Our results show that, even without any explicit spatial information about channels, our proposed architecture with channel permutation invariance and channel interactions work well in channel-agnostic multi-subject brain decoding.

Several recent studies proposed various methods for reconstructing natural images from human functional magnetic resonance imaging (fMRI) data. However, few studies have proposed reconstruction methods for electrophysiolgical brain activities such as electroencephalography (EEG) and electrocorticography (ECoG). To investigate whether natural images can be reconstructed from electrophysiological brain activities, we conducted a large-scale experiment on natural image reconstruction from ECoG signals using deep learning. We first recorded ECoG signals from two macaque monkeys while presenting diverse natural images. Then, we trained several deep learning models for reconstructing presented images from ECoG signals. Comparing reconstruction models, we find that models trained with an adversarial loss produced reconstructions that contain visible features in presented images. Furthermore, our results with downsampled ECoG signals show the importance of rich temporal dynamics in ECoG signals for image reconstruction. Our results indicate the possibility of reconstructing diverse natural images from electrophysiological brain activities using deep learning.