Distributed Graph Learning Cache Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed graph learning architectures, the need to store a complete backup of master data in each mirror node's memory leads to wasteful use of storage space, as all mirror data must be updated synchronously during data updates to maintain consistency between master and mirror nodes.
Innovation Solution
Implement a data access method where graph nodes are grouped based on dependency relationships, with cache space allocated from a common buffer to mirror nodes, allowing data access only after cache space allocation is completed, thereby eliminating the need for a complete backup of master data in each mirror node's memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a complete backup of master data is stored in the memory of each mirror node's graph learning device to ensure data consistency, then data consistency between master node and mirror node is maintained, but storage space of the graph learning device is wasted
Solution Approach 1:
The patent segments mirror data into two categories: frequently accessed data and infrequently accessed data. Frequently accessed data is cached in the common buffer, while infrequently accessed data remains in long-term storage. This segmentation allows the system to maintain data consistency for frequently accessed data while avoiding the need to store complete backups of all mirror data in memory, thus resolving the contradiction between data consistency and storage space utilization.
Solution Approach 2:
The patent applies local quality by providing different storage solutions for different types of data. The common buffer provides fast access for frequently accessed mirror data, while long-term storage handles less frequently accessed data. This localized optimization ensures that memory resources are allocated efficiently based on actual access patterns rather than uniformly storing all mirror data in memory.
2Reliability
If all mirror data is updated synchronously during data updates, then data consistency between master node and mirror node is ensured, but data access efficiency is reduced due to complete backup requirements
Solution Approach 1:
The patent performs preliminary action by pre-caching frequently accessed mirror data in the common buffer before it is needed for graph learning operations. This preliminary caching eliminates the need for synchronous updates of complete data backups during actual data access, thereby maintaining data consistency while significantly improving data access efficiency.
3Quantity of substance
If a common buffer is used to cache mirror node data with prioritized cache space allocation, then storage space utilization is improved, but system complexity increases due to node grouping and priority determination
Solution Approach 1:
The patent performs preliminary node grouping and priority determination before the actual caching process. By pre-establishing the dependency relationships and access priorities of graph nodes, the system simplifies the cache management process during runtime. This preliminary organization reduces the complexity of real-time cache allocation decisions while improving storage space utilization through targeted caching of high-priority mirror data.
Data Source
AI summary
In a data access method, graph nodes in the first graph learning device are grouped into a plurality of graph node groups with a priority. A priority of the graph node group is determined based on a graph node dependency relationship, and the graph node dependency relationship is used to reflect dependency of the graph node relative to the mirror node during graph learning. A mirror node on which each graph node group depends is determined based on the graph node dependency relationship; and cache space is allocated, from a common buffer of the first graph learning device based on the priority of the graph node group, to the mirror node on which each graph node group depends.