Training Graph Neural Networks with 1000 Layers

Guohao Li, Matthias Muller, Bernard Ghanem, Vladlen Koltun

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges. However, memory complexity has become a major obstacle when training deep GNNs for practical applications due to the immense number of nodes, edges, and intermediate activations. To improve the scalability of GNNs, prior works propose smart graph sampling or partitioning strategies to train GNNs with a smaller set of nodes or sub-graphs. In this work, we study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. We find that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets. Our models RevGNN-Deep (1001 layers with 80 channels each) and RevGNN-Wide (448 layers with 224 channels each) were both trained on a single commodity GPU and achieve an ROC-AUC of 87.74 ± 0.13 and 88.14 ± 0.15 on the ogbn-proteins dataset. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude.
Original languageEnglish (US)
Title of host publication38th International Conference on Machine Learning
StatePublished - 2021

Bibliographical note

KAUST Repository Item: Exported on 2021-09-30
Acknowledged KAUST grant number(s): OSR-CRG2019-4033
Acknowledgements: The authors thank Shaojie Bai, researchers at Intel ISL, the reviewers, and area chairs for their helpful suggestions. This project is partially funded by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research under Award No. OSR-CRG2019-4033.


Dive into the research topics of 'Training Graph Neural Networks with 1000 Layers'. Together they form a unique fingerprint.

Cite this