Graph Data Loading with Early Lock Release for Concurrency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Lock contention significantly affects the concurrent efficiency of graph data loading processes, especially in large-scale graph data processing scenarios, due to prolonged lock holding times during the loading process.
Innovation Solution
A method and apparatus that reduce lock holding time by determining a graph data loading identifier based on previous loaded data, releasing the lock, and storing the data at a corresponding storage location in the local memory, thereby improving concurrent efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If graph data is loaded concurrently from external storage to local memory, then loading speed and productivity are improved, but lock contention increases and concurrent efficiency deteriorates due to prolonged lock holding times
Solution Approach 1:
The patent calculates the storage location of graph data before releasing the lock, rather than after. This preliminary calculation of storage location using the loading identifier allows the lock to be released earlier, reducing lock holding time while maintaining data integrity. The storage location is determined in advance based on the loading identifier, enabling subsequent data writing to proceed without holding the lock.
2Productivity
If graph data is loaded concurrently from external storage to local memory, then loading throughput is improved, but lock contention increases and concurrent efficiency deteriorates
Solution Approach 1:
The patent performs the storage location calculation before releasing the lock, allowing multiple concurrent loading tasks to release locks independently and write data simultaneously. This eliminates the sequential dependency where one task must wait for another to complete the entire loading process, thereby improving concurrent efficiency while maintaining high throughput.
Data Source
AI summary
Embodiments of this specification provide a graph data loading method and apparatus. In the graph data loading method, in response to obtaining to-be-loaded graph data from the outside, a lock is obtained for the to-be-loaded graph data. A graph data loading identifier of the to-be-loaded graph data is determined based on a graph data loading identifier of previous loaded graph data stored in a graph data loading apparatus. Based on the graph data loading identifier of the to-be-loaded graph data, a corresponding storage location of the to-be-loaded graph data in a graph data storage space created in a local memory of a graph data processing device is determined. After the lock obtained for the to-be-loaded graph data is released, the to-be-loaded graph data is stored at the corresponding storage location in the graph data storage space.