Online Graph Data Import with External Sorting and Format Packaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph database systems face challenges in maintaining the ordering of an original data structure while ensuring efficient data import performance during the import process, leading to high network bandwidth load and resource consumption.
Innovation Solution
The method involves acquiring graph data in a first storage format, performing external sorting, packaging it into a second storage format compatible with the target graph database, and providing storage location information to ensure the graph database server imports the data with maintained ordering and high performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If graph data is imported using traditional methods, then data import can be completed, but the ordering of the original data structure is lost and import performance is poor
Solution Approach 1:
The patent performs external sorting on graph data before import, and packages the sorted data into the target storage format in advance. This preliminary organization of data ensures that when the data is imported to the graph database server, the ordering is already maintained, eliminating the need for post-import sorting operations and thus preserving data structure ordering while improving import performance.
2Stability of the object's composition
If external sorting is performed on graph data during import, then data structure ordering is maintained, but network bandwidth load and resource consumption increase
Solution Approach 1:
The patent divides the graph data into multiple sorted files based on vertex ID ranges, and packages each file separately with corresponding metadata. This segmentation allows for distributed processing and reduces the memory burden on the graph database server, enabling external sorting to be performed efficiently without overwhelming system resources or network bandwidth.
3Productivity
If graph data is packaged in a format compatible with the target graph database, then import efficiency is improved, but additional processing time is required
Solution Approach 1:
The patent packages the sorted graph data into the target storage format and generates corresponding metadata files before the import operation. By performing this format conversion and packaging in advance, the actual import process only requires loading pre-processed data, significantly improving import efficiency while the additional processing time is incurred before import rather than during it.
Data Source
AI summary
Embodiments of this specification provide methods and apparatuses for importing graph data online and graph database systems. In an example method for importing graph data online, graph data stored in a first storage format are acquired from a data source; external sorting is performed on the graph data to obtain sorted graph data; the sorted graph data are packaged based on a second storage format specified in a target graph database, to obtain a graph data file to be imported and a corresponding metadata file; and storage location information of the graph data file to be imported and the corresponding metadata file is provided to a graph database server, so that the graph database server imports the graph data file to the target graph database, and updates a metadata file corresponding to the target graph database based on the corresponding metadata file.