Leaky Tiling Activations: A Simple Approach to Learning Sparse Representations Online
international conference on learning representations(2019)
摘要
Interference is a known problem when learning in online settings, such as continual learning or reinforcement learning. Interference occurs when updates, to improve performance for some inputs, degrades performance for others. Recent work has shown that sparse representations---where only a small percentage of units are active---can significantly reduce interference. Those works, however, relied on relatively complex regularization or meta-learning approaches, that have only been used offline in a pre-training phase. In our approach, we design an activation function that naturally produces sparse representations, and so is much more amenable to online training. The idea relies on the simple approach of binning, but overcomes the two key limitations of binning: zero gradients for the flat regions almost everywhere, and lost precision---reduced discrimination---due to coarse aggregation. We introduce a Leaky …
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络