Brain-grounding of semantic vectors improves neural decoding of visual stimuli
arxiv(2024)
摘要
Developing algorithms for accurate and comprehensive neural decoding of
mental contents is one of the long-cherished goals in the field of neuroscience
and brain-machine interfaces. Previous studies have demonstrated the
feasibility of neural decoding by training machine learning models to map brain
activity patterns into a semantic vector representation of stimuli. These
vectors, hereafter referred as pretrained feature vectors, are usually derived
from semantic spaces based solely on image and/or text features and therefore
they might have a totally different characteristics than how visual stimuli is
represented in the human brain, resulting in limiting the capability of brain
decoders to learn this mapping. To address this issue, we propose a
representation learning framework, termed brain-grounding of semantic vectors,
which fine-tunes pretrained feature vectors to better align with the neural
representation of visual stimuli in the human brain. We trained this model this
model with functional magnetic resonance imaging (fMRI) of 150 different visual
stimuli categories, and then performed zero-shot brain decoding and
identification analyses on 1) fMRI and 2) magnetoencephalography (MEG).
Interestingly, we observed that by using the brain-grounded vectors, the brain
decoding and identification accuracy on brain data from different neuroimaging
modalities increases. These findings underscore the potential of incorporating
a richer array of brain-derived features to enhance performance of brain
decoding algorithms.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要