Reversible Graph Neural Network Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graph neural networks (GNNs) face memory bottlenecks when training on large graphs due to their limited parameter capacity and memory complexity, which restricts their ability to learn effectively on huge datasets, and existing solutions like mini-batch training and partitioning introduce additional hyperparameters and sub-optimality.
Innovation Solution
The development of graph neural networks with constant memory complexity using reversible connections, group convolutions, weight tying, and deep equilibrium model architectures, enabling the training of very deep and overparameterized models with minimal memory consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep graph neural networks are trained on large graphs, then model performance and accuracy are improved, but memory complexity and computational resources are excessively consumed
Solution Approach 1:
The patent applies segmentation by dividing the graph into multiple subgraphs and training the neural network on these smaller partitions rather than the entire large graph at once. This allows the model to achieve deep architecture benefits for improved accuracy while reducing memory complexity to manageable levels through manageable data partitions.
Solution Approach 2:
The patent introduces an intermediary mechanism using a two-stage training process with a pre-trained model that provides initial weights. This intermediary approach allows transfer learning where the pre-trained model serves as a mediator, enabling fine-tuning on specific subgraphs without requiring the full computational resources needed for training from scratch on large graphs.
2Use of energy by moving object
If mini-batch training or partitioning is used to reduce memory consumption, then memory complexity is reduced, but additional hyperparameters are introduced and optimal performance is compromised
Solution Approach 1:
The patent applies self-service by using the pre-trained model's weights as automatic initialization for the fine-tuned model on subgraphs. This self-service mechanism eliminates the need for manual hyperparameter tuning for weight initialization, as the pre-trained weights provide a ready-made starting point that adapts to the subgraph data without requiring additional complex configuration.
3Adaptability or versatility
If the number of model parameters is increased to handle large graphs, then parameter capacity is improved, but memory requirements and training difficulty increase
Solution Approach 1:
The patent applies partial action by training the neural network on a subset of the graph data (subgraphs) rather than the complete graph. This partial training approach allows the model to achieve sufficient parameter capacity for the specific subgraph task at hand, making training feasible while avoiding the excessive computational burden of processing the entire large graph with full parameter capacity.
Data Source
AI summary
Systems, apparatuses and methods may provide for technology that trains a reversible graph neural network (GNN) by partitioning an input vertex feature matrix into a plurality of groups, generating, via a block of the reversible GNN, outputs for the plurality of groups based on an adjacency matrix and an edge feature matrix, wherein the outputs are generated during one or more forward propagations, conducting a reconstruction of the input feature matrix during one or more backward propagations, and excluding the adjacency matrix and the edge feature matrix from the reconstruction. The technology also trains a deep equilibrium GNN.