Graph Data Partitioning for MapReduce I/O Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face performance issues when applying MapReduce to structured data due to an unoptimal storage layer, leading to inefficient data processing and high I/O operations, particularly in large-scale data warehousing queries.
Innovation Solution
The method involves partitioning data into disjoint partitions based on vertexes in a graph data model, assigning partitions to nodes, and processing queries by logically connecting vertexes through edges, allowing for parallel processing and efficient data management using Apache Hadoop.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If MapReduce is used to process structured data, then scalability and parallel processing capability are improved, but I/O operations and data movement increase significantly
Solution Approach 1:
The patent segments data into disjoint partitions based on graph vertexes and assigns them to different nodes. This segmentation allows parallel processing while minimizing cross-node data movement, as each partition is self-contained and processed independently on its assigned node.
Solution Approach 2:
The patent performs preliminary data partitioning and node assignment before query execution. By pre-organizing data into partitions and assigning them to nodes based on graph structure, the system reduces the need for data movement during actual query processing, thereby reducing I/O operations.
2Speed
If data is partitioned and distributed across multiple nodes, then processing speed is improved, but data movement and network transfer increase
Solution Approach 1:
The patent applies local quality by ensuring that each node processes only the data partitions assigned to it, based on graph vertex assignments. This localized processing minimizes network data movement while maintaining parallel execution speed, as each node operates independently on its local data subset.
Solution Approach 2:
By segmenting the graph data into disjoint partitions and distributing them across nodes, the system enables parallel query execution while minimizing inter-node data transfer. Each partition is self-contained, reducing the need for data movement during processing.
3Device complexity
If conventional storage layers are used with MapReduce, then system simplicity is maintained, but query execution efficiency deteriorates
Solution Approach 1:
The patent introduces dynamic data partitioning and node assignment mechanisms that adapt to graph structure and query patterns. This dynamic approach improves query execution efficiency by optimizing data placement and processing strategies while maintaining reasonable system complexity through automated management.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
System, method and computer program product for processing a query are disclosed. Query processing includes partitioning the stored data into a plurality of partitions based on at least one vertex in the plurality of vertexes, storing at least another triple in the plurality of triples on the at least one node, assigning, based on the triple containing the at least one vertex, at least one partition in the plurality of partitions corresponding to the triple to at least one node in the plurality of nodes, and processing, based on the assigning, the query by processing the plurality of partitions.