CGD-Based Graph Continual Learning for Memory-Efficient Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph continual learning methods face challenges such as high memory storage requirements, catastrophic forgetting of previous patterns, and scalability issues when dealing with time-evolving graphs, leading to poor performance on previous prediction tasks.
Innovation Solution
The use of condensed graph distributions (CGDs) and a stochastic memory buffer for graph continual learning, where the primary GNN model is trained with selective sampling from stored CGDs to fine-tune predictions, employing a majority voting technique for improved scalability and reduced memory overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If previous graphs are stored and processed with new incoming training samples, then performance degradation is avoided, but large memory storage is required
Solution Approach 1:
The patent extracts only the essential statistical information (mean and covariance) from the complete graph data to create condensed graph distributions. This extraction allows the model to retain knowledge of previous tasks without storing the actual graph structures, significantly reducing memory requirements while maintaining performance on previous tasks.
Solution Approach 2:
Instead of storing complete graphs and extracting features when needed, the patent inverts the approach by pre-computing and storing only the statistical distributions (mean and covariance) of graph features. This inversion transforms the storage burden from large graph structures to compact statistical parameters.
2Stability of the object's composition
If a regularizer is used to restrict updates of weights, then catastrophic forgetting is prevented, but poor performance is observed on previous prediction tasks
Solution Approach 1:
The patent introduces condensed graph distributions as an intermediary between previous task data and the current model. These distributions serve as a mediator that provides guidance during weight updates, allowing the model to adapt to new tasks while maintaining performance on previous tasks without relying on regularizers that restrict weight updates.
3Adaptability or versatility
If model growing techniques are applied, then the model can handle increasing number of tasks, but linear growth of model parameters occurs making the technique difficult to scale
Solution Approach 1:
The patent changes the parameters from complete graph structures to condensed statistical distributions (mean and covariance). This parameter transformation allows the model to handle an increasing number of tasks without linear growth in complexity, as the condensed parameters remain compact regardless of the number of tasks.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In an embodiment, operations include receiving a first graph associated with a first task following graph learning tasks including a sequence of second graphs. A set of sample graphs is selected from a set of condensed graph distributions (CGDs) associated with the graph learning tasks. A set of statistics associated with the set of CGDs is updated, based on one or more auxiliary graph neural network (GNN) models, the first graph, and the set of sample graphs. A first CGD associated with the first task is learned. A plurality of sample graphs is re-selected from the first CGD and the set of CGDs. A first loss corresponding to a prediction error associated with a downstream prediction task of the primary GNN model is determined. A prediction result associated with the downstream prediction task is generated by the primary GNN model, based on the first loss.