FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Training
Proceedings of the VLDB Endowment(2023)
摘要
A key performance bottleneck when training graph neural network (GNN) models
on large, real-world graphs is loading node features onto a GPU. Due to limited
GPU memory, expensive data movement is necessary to facilitate the storage of
these features on alternative devices with slower access (e.g. CPU memory).
Moreover, the irregularity of graph structures contributes to poor data
locality which further exacerbates the problem. Consequently, existing
frameworks capable of efficiently training large GNN models usually incur a
significant accuracy degradation because of the currently-available shortcuts
involved. To address these limitations, we instead propose FreshGNN, a
general-purpose GNN mini-batch training framework that leverages a historical
cache for storing and reusing GNN node embeddings instead of re-computing them
through fetching raw features at every iteration. Critical to its success, the
corresponding cache policy is designed, using a combination of gradient-based
and staleness criteria, to selectively screen those embeddings which are
relatively stable and can be cached, from those that need to be re-computed to
reduce estimation errors and subsequent downstream accuracy loss. When paired
with complementary system enhancements to support this selective historical
cache, FreshGNN is able to accelerate the training speed on large graph
datasets such as ogbn-papers100M and MAG240M by 3.4x up to 20.5x and reduce the
memory access by 59
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要