Map-Reduce Connected Component Algorithm Using Dynamic Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing connected components in large distributed graphs is time and cost prohibitive due to the dependency of hashing functions on graph size and the number of messages sent between machines during Map-Reduce rounds.
Innovation Solution
Alternating rounds of Map-Reduce between Hash-Greater-to-Min and Hash-Lesser-to-Min functions, and using an in-memory table to reduce messaging between nodes and machines, optimizing load-balancing for highly connected nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If traditional Hash-to-Min function is used in each round of Map-Reduce, then the algorithm is simple to implement, but the computing time is excessively long (2 to 4.5 times slower)
Solution Approach 1:
The patent dynamically alternates between two different hashing functions (Hash-Greater-to-Min and Hash-Lesser-to-Min) across different Map-Reduce rounds, rather than using a static single function. This dynamic switching optimizes the distribution of keys to reducers, balancing the load and significantly improving computing speed while maintaining algorithmic simplicity
Solution Approach 2:
The patent changes the hashing function parameter (switching between Hash-Greater-to-Min and Hash-Lesser-to-Min) based on the round number, where even rounds use one function and odd rounds use the other. This parameter change adapts the system to achieve better performance without complex implementation
2Reliability
If Map-Reduce continues without using in-memory table, then the algorithm maintains consistency, but the number of messages sent between machines is excessively large
Solution Approach 1:
The patent performs a preliminary action by creating an in-memory table after a predetermined number of Map-Reduce rounds (e.g., 2 rounds). This table stores active cluster identifiers and their associated nodes, which is built in advance to eliminate the need for continuous messaging in subsequent rounds, thereby reducing communication overhead while maintaining algorithmic consistency
Solution Approach 2:
The in-memory table acts as an intermediary between the Map-Reduce computation and the final result. Instead of continuing to send messages between machines, the computation references this intermediate table structure, which mediates the remaining rounds and eliminates further messaging while preserving correctness
3Ease of manufacture
If highly connected nodes are not optimized, then the algorithm is simpler, but the maximum number of values received by a reducer node becomes unmanageably large
Solution Approach 1:
The patent applies different hashing strategies locally based on node characteristics. For highly connected nodes, the alternating hashing functions distribute their keys more evenly across reducers, preventing any single reducer from receiving an unmanageable number of values. This local optimization addresses the specific problem of high-degree nodes without complicating the overall algorithm
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for improving the time and cost to calculate connected components in a distributed graph are disclosed. One method includes reducing a quantity of map-reduce rounds used to determine a cluster assignment for a node in a large distributed graph by alternating between two hashing functions in the map stage of a map-reduce round and storing the cluster assignment for the node in a memory. Another method includes reducing a quantity of messages sent during map-reduce rounds by performing a predetermined quantity of rounds to generate, for each node, a set of potential cluster assignments, generating a data structure in memory to store a mapping between each node and its potential cluster assignment, and using the data structure during remaining map-reduce rounds, wherein the remaining map-reduce rounds do not send messages between nodes. The method can also include storing the cluster assignment for the node in a memory.