Property Graph Partitioning for Parallel Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Property graph ingestion is slow due to its storage model, and attempting to hasten this process through multiple threads can result in an inconsistent graph that fails to meet constraint requirements.
Innovation Solution
The method involves partitioning and parallel loading of property graphs with constraints using polynomial time partition algorithms, where the graph is divided into sub-graphs and allocated to multiple threads to ensure consistency and adherence to constraints during the loading process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple threads are used to hasten property graph ingestion, then ingestion speed is improved, but graph consistency and constraint fulfillment deteriorate
Solution Approach 1:
The property graph is divided into multiple partitions, each handled by a separate thread. This segmentation allows parallel processing while maintaining consistency within each partition, resolving the contradiction between speed and consistency by isolating concurrent access to discrete graph segments.
Solution Approach 2:
A constraint satisfaction mechanism acts as an intermediary between parallel thread operations and the final graph state. This intermediary ensures that constraint violations are detected and resolved, maintaining graph consistency even during parallel ingestion operations.
2Productivity
If multiple threads are used to hasten property graph ingestion, then ingestion speed is improved, but constraint fulfillment deteriorates
Solution Approach 1:
The system implements feedback mechanisms where constraint violations detected during parallel ingestion trigger corrective actions. This feedback loop ensures that constraint fulfillment is maintained by detecting and resolving violations while allowing parallel processing to continue.
Solution Approach 2:
Constraint checking and partitioning are performed as preliminary actions before the main parallel ingestion process. By pre-establishing valid partitions and constraint rules, the system enables faster parallel loading while ensuring constraints are inherently satisfied, avoiding the need for corrective actions during ingestion.
3Productivity
If polynomial time partition algorithms are used to partition the graph, then parallel loading efficiency is improved, but processing complexity increases
Solution Approach 1:
The system changes the parameter of partitioning complexity by selecting polynomial time algorithms instead of optimal but computationally expensive algorithms. This parameter change achieves a practical balance where partitions are sufficiently optimized for parallel loading while keeping the preprocessing complexity manageable.
Data Source
AI summary
Methods, systems, and computer program products for partitioning and parallel loading of property graphs with constraints are provided herein. A computer-implemented method includes obtaining graph-related input data and corresponding constraint data, wherein the graph-related input data and corresponding constraint data are at least one of user-defined and input data model-based; generating at least one in-memory graph based at least in part on processing at least a portion of the obtained graph-related input data; partitioning the at least one in-memory graph into two or more sub-graphs by processing the at least one in-memory graph using one or more polynomial time partition algorithms; and generating at least one property graph by allocating, at least a portion of the two or more partitioned sub-graphs which satisfy the obtained constraint data, to two or more threads that run in parallel.