Distributed Graph Storage Manager for Hierarchical Block Placement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed systems face challenges in efficiently processing large-scale graph data due to high storage resource requirements and the need for effective management and storage of graph data during tasks such as graph computing and graph mining.
Innovation Solution
A distributed graph data processing system with a computing node architecture that includes a computing unit and a persistent storage unit, utilizing a storage manager to divide graph data into blocks, determine key values, and store them in a hierarchical storage structure comprising memory, persistent storage medium, and hard disk, supporting fault tolerance through persistent storage of intermediate state data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If graph data is stored in a distributed system with high storage resource requirements, then the system can handle large-scale graph data, but the storage resource consumption increases
Solution Approach 1:
The patent segments graph data into graph data blocks and organizes them into a hierarchical storage structure with multiple levels (memory, persistent storage medium, hard disk). This segmentation allows the system to store large-scale graph data while managing storage resources efficiently by placing different data blocks at appropriate storage levels based on access patterns and importance.
2Speed
If graph data is divided into blocks with key values and storage location information, then data retrieval efficiency improves, but the system complexity increases
Solution Approach 1:
The patent introduces a storage manager as an intermediary component that handles the complexity of graph data block management. The storage manager is deployed in the computing unit and automatically manages the division of graph data into blocks, assignment of key values, determination of storage locations, and maintenance of the hierarchical storage structure. This intermediary absorbs the management complexity while providing efficient data retrieval services to the computing unit.
3Productivity
If a hierarchical storage structure with memory, persistent storage medium, and hard disk is used, then processing efficiency is maintained, but the device complexity increases
Solution Approach 1:
The patent implements a universal hierarchical storage structure that can accommodate different types of storage media (memory, persistent storage medium, hard disk) within a unified management framework. The storage manager provides multi-functional capabilities by managing diverse storage devices through a common interface and methodology, allowing the system to maintain high processing efficiency while avoiding the complexity that would arise from managing multiple independent storage systems.
4Reliability
If persistent storage unit is used to support fault tolerance, then system reliability improves, but the storage resource requirements increase
Solution Approach 1:
The patent applies local quality by differentiating the storage requirements of different graph data blocks. The hierarchical storage structure assigns different storage locations and persistence levels to different data blocks based on their specific characteristics, access patterns, and importance to fault tolerance. This allows the system to provide fault tolerance for critical data while using more economical storage for less critical data, thereby improving reliability without uniformly increasing storage resource requirements across all data.
Data Source
AI summary
The present disclosure provides a distributed graph data processing system, method, apparatus and device, and a storage medium. The system includes: a computing node in a distributed architecture, wherein the computing node includes a computing unit and a persistent storage unit that are in a corresponding relationship, and a storage manager is deployed in each computing unit. The storage manager divides the graph data adjacency list into graph data blocks, determines a key value corresponding to each of the graph data blocks, and then records and stores a corresponding relationship between key values of the graph data blocks and storage location information.