Knowledge Graph Embedding Quantization for Memory-Saving KGNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training knowledge graph neural networks (KGNNs) requires significant computing resources and memory due to large data volumes, especially for multi-hop relations, and distributed training frameworks face challenges with latency and loss of relational connections.
Innovation Solution
Implement a method involving quantization and dequantization of knowledge graph embeddings using equations to reduce memory footprint, allowing storage on GPUs and enabling efficient backpropagation with gradients determined from dequantized embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed resources are used to store and access KG embeddings, then memory capacity is increased, but training latency increases and multi-hop relational connections are lost
Solution Approach 1:
The patent segments the knowledge graph into multiple partitions that can be distributed across different computing devices. Each partition contains a subset of entities and relations, allowing parallel processing while maintaining the ability to traverse multi-hop relationships across partition boundaries through a unified embedding space.
Solution Approach 2:
The patent introduces an intermediary communication mechanism that enables exchanging embedding information between distributed partitions. This intermediary layer allows nodes in different partitions to access embeddings from neighboring partitions without requiring full data replication, thus reducing latency while maintaining memory capacity.
2Quantity of substance
If KG is partitioned for distributed processing, then memory capacity is increased, but multi-hop relational connections are broken
Solution Approach 1:
The knowledge graph is segmented into partitions based on entity connectivity patterns, ensuring that multi-hop relationships are preserved across partition boundaries. The segmentation strategy maintains the structural integrity of relational paths by grouping entities that frequently appear together in multi-hop sequences.
Solution Approach 2:
The patent creates a universal embedding space that functions across all partitions, allowing embeddings from different partitions to be combined and processed together. This universal space enables multi-hop relationship learning by allowing messages to propagate across partition boundaries while maintaining consistent semantic representations.
3Measurement precision
If full precision KG embeddings are stored, then training accuracy is maintained, but memory requirements increase
Solution Approach 1:
The patent dynamically changes the precision parameter of embeddings based on their usage patterns and importance in the training process. Frequently accessed or critical embeddings are maintained at full precision, while less critical embeddings use lower precision representations, optimizing the trade-off between memory usage and training accuracy.
Data Source
AI summary
Systems, methods, and computer program products are provided for saving memory during training of knowledge graph neural networks. The method includes receiving a training dataset including a first set of knowledge graph embeddings associated with a plurality of entities for a first layer of a knowledge graph, inputting the training dataset into a knowledge graph neural network to generate at least one further set of knowledge graph embeddings associated with the plurality of entities for at least one further layer of the knowledge graph, quantizing the at least one further set of knowledge graph embeddings to provide at least one set of quantized knowledge graph embeddings, storing the at least one set of quantized knowledge graph embeddings in a memory, and dequantizing the at least one set of quantized knowledge graph embeddings to provide at least one set of dequantized knowledge graph embeddings.