EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations
ICLR 2024(2023)
摘要
Equivariant Transformers such as Equiformer have demonstrated the efficacy of
applying Transformers to the domain of 3D atomistic systems. However, they are
limited to small degrees of equivariant representations due to their
computational complexity. In this paper, we investigate whether these
architectures can scale well to higher degrees. Starting from Equiformer, we
first replace SO(3) convolutions with eSCN convolutions to efficiently
incorporate higher-degree tensors. Then, to better leverage the power of higher
degrees, we propose three architectural improvements – attention
re-normalization, separable S^2 activation and separable layer normalization.
Putting this all together, we propose EquiformerV2, which outperforms previous
state-of-the-art methods on large-scale OC20 dataset by up to 9% on forces,
4% on energies, offers better speed-accuracy trade-offs, and 2×
reduction in DFT calculations needed for computing adsorption energies.
Additionally, EquiformerV2 trained on only OC22 dataset outperforms GemNet-OC
trained on both OC20 and OC22 datasets, achieving much better data efficiency.
Finally, we compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M
datasets to better understand the performance gain brought by higher degrees.
更多查看译文
关键词
equivariant neural networks,graph neural networks,computational physics,transformer networks
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要