Language Modeling Using Tensor Trains
arxiv(2024)
摘要
We propose a novel tensor network language model based on the simplest tensor
network (i.e., tensor trains), called `Tensor Train Language Model' (TTLM).
TTLM represents sentences in an exponential space constructed by the tensor
product of words, but computing the probabilities of sentences in a
low-dimensional fashion. We demonstrate that the architectures of Second-order
RNNs, Recurrent Arithmetic Circuits (RACs), and Multiplicative Integration RNNs
are, essentially, special cases of TTLM. Experimental evaluations on real
language modeling tasks show that the proposed variants of TTLM (i.e.,
TTLM-Large and TTLM-Tiny) outperform the vanilla Recurrent Neural Networks
(RNNs) with low-scale of hidden units. (The code is available at
https://github.com/shuishen112/tensortrainlm.)
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要