In-Memory Graph Construction via Parallel Node Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph data migration from a database to an analytic framework is time-consuming and not user-friendly, especially for large datasets, due to the inefficiencies in existing file-based migration methods.
Innovation Solution
The process involves multiple phases such as node loading, merging, property construction, edge loading, merging, and property construction, where node and edge data are loaded and merged in parallel using a priority queue system to create an efficient in-memory representation of a graph, utilizing a mapping system to index nodes and edges for faster data retrieval and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If file-based graph data migration is used, then data can be transferred from database to analytic framework, but the migration process takes significant amount of time
Solution Approach 1:
The patent segments the graph data migration process into distinct phases: node loading phase, node merging phase, edge loading phase, and edge merging phase. Each phase processes specific components (nodes or edges) separately, allowing for optimized handling and parallel processing where applicable, thereby reducing overall migration time while maintaining data integrity
Solution Approach 2:
The patent performs preliminary actions by pre-processing and organizing graph data before the actual migration. This includes preparing node and edge data in advance, establishing data structures, and setting up the analytic framework environment beforehand, which significantly reduces the time required during the actual migration execution
2Speed
If in-memory computation is adopted for fast graph analysis, then computation speed is improved, but data migration time increases significantly
Solution Approach 1:
The patent changes the parameters of data representation and organization during migration. It transforms graph data into optimized in-memory structures with specific indexing schemes and data layouts that are tailored for fast computation. This parameter optimization ensures that while migration takes place, the resulting in-memory representation enables significantly faster graph analysis operations
3Loss of time
If graph data is loaded and merged in parallel phases, then migration time is reduced, but system complexity increases
Solution Approach 1:
The patent divides the complex parallel processing task into manageable segments: node loading, node merging, edge loading, and edge merging phases. Each segment handles specific data types and operations, making the overall complex process more controllable and easier to implement while still achieving parallel processing benefits
Solution Approach 2:
The patent introduces intermediary data structures and temporary storage mechanisms that facilitate parallel processing. These intermediaries act as buffers and coordinators between different parallel processing threads, managing data flow and synchronization without requiring complex direct coordination between all processing components
Data Source
AI summary
Techniques for efficiently loading graph data into memory are provided. A plurality of node ID lists are retrieved from storage. Each node ID list is ordered based on one or more order criteria, such as node ID, and is read into memory. A new list of node IDs is created in memory and is initially empty. From among the plurality of node ID lists, a particular node ID is selected based on the one or more order criteria, removed from the node ID list where the particular node ID originates, and added to the new list. This process of selecting, removing, and adding continues until no more than one node ID list exists, other than the new list. In this way, the retrieval of the plurality of node ID lists from storage may be performed in parallel while the selecting and adding are performed sequentially.