Online Graph Data Import Using External Sorting and Metadata Merge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph database systems face challenges in maintaining the ordering of an original data structure while ensuring efficient data import performance during the import process, leading to high network bandwidth load and resource occupation.
Innovation Solution
The method involves acquiring graph data in a first storage format, performing external sorting, packaging it into a second storage format specified by the target graph database, and providing storage location information to ensure the graph database server imports the data with maintained ordering and high performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If data is imported sequentially into an ordered storage engine to maintain data structure ordering, then data structure ordering is maintained, but import performance deteriorates due to high network bandwidth load and resource occupation
Solution Approach 1:
The patent divides the import process into two independent phases: offline pre-processing (sorting and packaging) and online import (merging). This segmentation allows the resource-intensive sorting operation to be performed separately from the import operation, enabling parallel execution and eliminating the trade-off between maintaining ordering and achieving high import performance.
Solution Approach 2:
The patent performs external sorting and data packaging in advance during an offline phase, before the actual import operation. By pre-sorting the data and packaging it into the target storage format, the system eliminates the need to perform sorting during the online import phase, thus maintaining data structure ordering while achieving high import performance.
2Stability of the object's composition
If external sorting is performed on large-scale graph data, then data structure ordering is maintained, but memory resources are insufficient for in-memory sorting
Solution Approach 1:
The patent transitions the sorting operation from the memory dimension to the disk dimension by implementing external sorting. Instead of attempting to load all data into memory for sorting, the system performs sorting operations on disk-based data structures, utilizing disk storage capacity to overcome memory limitations while still achieving the required data ordering.
3Productivity
If data is pre-processed offline including sorting and packaging, then import performance is improved, but storage space is increased due to generating multiple data files
Solution Approach 1:
The patent combines multiple pre-processed data files into a single merged data file during the online import phase. By merging the sorted and packaged data files generated during offline pre-processing, the system achieves the benefits of pre-processing (improved import performance) while eliminating the storage overhead of maintaining multiple separate data files.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
Embodiments of this specification provide methods and apparatuses for importing graph data online and graph database systems. In the methods for importing graph data online, graph data stored in a first storage format are acquired from a data source; external sorting is performed on the graph data to obtain sorted graph data; the sorted graph data are packaged based on a second storage format specified in a target graph database, to obtain a graph data file to be imported and a corresponding metadata file; and storage location information of the graph data file to be imported and the corresponding metadata file is provided to a graph database server, so that the graph database server imports the graph data file to the target graph database, and updates a metadata file corresponding to the target graph database based on the obtained metadata file.