The Trifecta: Three simple techniques for training deeper Forward-Forward networks
CoRR(2023)
摘要
Modern machine learning models are able to outperform humans on a variety of
non-trivial tasks. However, as the complexity of the models increases, they
consume significant amounts of power and still struggle to generalize
effectively to unseen data. Local learning, which focuses on updating subsets
of a model's parameters at a time, has emerged as a promising technique to
address these issues. Recently, a novel local learning algorithm, called
Forward-Forward, has received widespread attention due to its innovative
approach to learning. Unfortunately, its application has been limited to
smaller datasets due to scalability issues. To this end, we propose The
Trifecta, a collection of three simple techniques that synergize exceptionally
well and drastically improve the Forward-Forward algorithm on deeper networks.
Our experiments demonstrate that our models are on par with similarly
structured, backpropagation-based models in both training speed and test
accuracy on simple datasets. This is achieved by the ability to learn
representations that are informative locally, on a layer-by-layer basis, and
retain their informativeness when propagated to deeper layers in the
architecture. This leads to around 84\% accuracy on CIFAR-10, a notable
improvement (25\%) over the original FF algorithm. These results highlight the
potential of Forward-Forward as a genuine competitor to backpropagation and as
a promising research avenue.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要