Graph State Data Management via Key-Value Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph computing technologies face challenges in managing graph state data effectively, particularly in real-time scenarios where data fault tolerance and scalability are crucial, as they often lack a clear separation of computation and storage, leading to inefficiencies in data management.
Innovation Solution
A method and apparatus for graph state data management that decouples graph state management from graph computing, utilizing key-value (kv) data encoding, sorting, and storage in a file system with a memory index to enable efficient storage and retrieval of vertex and edge data, allowing for larger-scale data management and aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If graph state data is stored in memory or cache of the graph computing engine, then data access speed is improved, but device complexity and scalability deteriorate
Solution Approach 1:
The patent divides the storage system into two independent segments: a file storage system for persistent graph state data and a memory index for rapid lookup. The computing engine interacts with the memory index rather than directly managing storage, separating computation from storage management. This segmentation allows fast access through the memory index while the file storage system handles scalability without increasing computing engine complexity.
Solution Approach 2:
The patent introduces a memory index as an intermediary layer between the graph computing engine and the file storage system. The memory index stores key-value mappings that enable rapid data location, while the actual graph state data resides in the file storage system. This intermediary structure provides fast access speeds without requiring the computing engine to directly manage large-scale storage, thus maintaining low device complexity while achieving high scalability.
2Adaptability or versatility
If graph state data is stored in external file storage system, then scalability is improved, but data access speed deteriorates
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing key-value mappings in the memory index before actual data access occurs. When graph state data needs to be accessed, the system first queries the memory index to obtain the logical address, then directly accesses the file storage system using this address. This preliminary indexing action enables fast retrieval from external storage without sacrificing scalability, as the heavy lifting of data organization is done in advance.
3Ease of operation
If computation and storage are integrated in the graph computing engine, then ease of operation is improved, but productivity and scalability deteriorate
Solution Approach 1:
The patent segments the system into distinct functional modules: the graph computing engine focuses on computation while the file storage system handles persistent storage, and the memory index manages data location. This segmentation allows each component to be optimized independently, improving overall productivity. The computing engine can process data without being burdened by storage management complexity, while the storage system can scale independently to handle larger datasets.
Data Source
AI summary
Embodiments of this specification provide a graph state data management method and apparatus. The method includes: encoding, after acquiring batch graph state data from a graph computing engine, each piece of graph state data in the batch graph state data into kv data; sorting the kv data based on a key of the kv data to form kv list data, where in the kv list data, each key corresponds to one or more values; next, sequentially writing values of the kv list data into a data file in a file storage system, and recording a corresponding logical address of each key in the data file; and then, maintaining a memory index of the batch graph state data in a memory of a graph state management device, where the maintained memory index is used to reflect an index relationship between a key and a corresponding logical address.