Graph Key-Value Layout With Prefetching for Faster Data Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing scale and complexity of graph data analysis lead to frequent hard disk reading operations, impacting overall performance due to inefficient storage and retrieval methods.
Innovation Solution
A key-value pair storage method that organizes graph composition elements based on category identifiers, ensuring same-category data is stored adjacently, and a graph data prefetching method that reads related data into cache for improved efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If graph data is stored using traditional storage methods, then the system can handle large scale graph data, but frequent hard disk reading operations occur which impacts performance
Solution Approach 1:
The patent implements a prefetching mechanism that pre-loads graph composition element data into memory before it is actually needed for processing. When a node is being processed, the system automatically prefetches its associated edge data and neighboring node data into the prefetch buffer, so that when this data is needed, it is already available in memory rather than requiring hard disk reading operations. This preliminary action significantly reduces I/O wait time and improves overall processing throughput.
Solution Approach 2:
The patent introduces a prefetch buffer as an intermediary layer between the hard disk storage and the processing units. This buffer serves as a mediator that caches frequently accessed graph data (nodes, edges, and their relationships) in memory, reducing the frequency of hard disk reads. The buffer management unit acts as an intermediary that intelligently manages what data to prefetch and when, optimizing the balance between memory usage and I/O operations.
2Productivity
If graph data is stored in a centralized and orderly way based on category identifiers, then cache hit rate improves, but storage organization complexity increases
Solution Approach 1:
The patent segments graph data into distinct categories based on composition elements (nodes, edges, and their attributes). Each category is assigned a specific storage region in the key-value store, with keys structured to reflect these categories. This segmentation allows the prefetching mechanism to efficiently prefetch entire categories of related data into memory, improving cache hit rates by ensuring that when one piece of data is accessed, its category mates are likely to be needed soon and are already available in memory.
Solution Approach 2:
The patent changes the organizational parameter of storage from traditional flat or hierarchical structures to a category-based arrangement using key-value pairs with structured keys. The key format incorporates category identifiers that enable efficient grouping and retrieval of related data. This parameter change allows the system to optimize for sequential access patterns within categories, improving prefetching effectiveness and cache utilization while maintaining manageable complexity through consistent key formatting rules.
Data Source
AI summary
One or more implementations of this specification provides key-value pair storage methods and devices for graph data. In an implementation, a key-value pair storage method includes: obtaining key-value pairs of graph composition elements, wherein values of the key-value pairs correspond to graph data of graph composition elements, keys of the key-value pairs each comprise identifier fields, wherein the identifier fields comprise a category identifier of a corresponding graph composition element of the graph composition elements, wherein the graph composition elements are at least one of nodes or edges, and storing the key-value pairs in order based on an arrangement sequence of category identifiers comprised in the keys.