Some RNN Variants
user-5ebe27fd4c775eda72abcdc7(2017)
摘要
Page 1. Some RNN Variants Arun Mallya Best viewed with Computer Modern fonts installed Page 2. Outline • Why Recurrent Neural Networks (RNNs)? • The Vanilla RNN unit • The RNN forward pass • Backpropagation refresher • The RNN backward pass • Issues with the Vanilla RNN • The Long Short-Term Memory (LSTM) unit • The LSTM Forward & Backward pass • LSTM variants and tips – Peephole LSTM – GRU Page 3. The Vanilla RNN Cell 3 ht xt ht-1 h t = tanhW x t h t−1 ⎛ ⎝⎜ ⎞ ⎠ W Page 4. The Vanilla RNN Forward 4 h1 x1 h0 C 1 y1 h2 x2 h1 C 2 y2 h3 x3 h2 C 3 y3 h t = tanhW x t h t−1 ⎛ ⎝⎜ ⎞ ⎠ y t = F(h t ) C t = Loss(y t ,GT t ) Page 5. The Vanilla RNN Forward 5 h1 x1 h0 C 1 y1 h2 x2 h1 C 2 y2 h3 x3 h2 C 3 y3 h t = tanhW x t h t−1 ⎛ ⎝⎜ ⎞ ⎠ y t = F(h t ) C t = Loss(y t ,GT t ) indicates shared weights Page 6. The Vanilla RNN Backward 6 h1 x1 h0 C 1 y1 h2 x2 h1 C 2 y2 h3 x3 h2 C 3 y3 h t = tanhW x t h t−1 …
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络