Multi-Pass Data Shuffle for Scalable Repartitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Shuffle operations in distributed networks face scalability issues as the number of sinks increases, leading to quadratic growth in the number of operations required, which can exceed the capacity of available resources.
Innovation Solution
Implementing a multi-pass shuffle approach that divides the shuffle operations into multiple passes, reducing the number of sources read by each sink and optimizing the distribution scheme to improve efficiency and reduce overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional single-pass shuffle operations are used to distribute data from multiple sources to multiple sinks, then the shuffle operation can be completed in a single pass, but the number of operations increases quadratically as the number of sinks increases, leading to scalability issues and increased overhead
Solution Approach 1:
The shuffle operation is divided into multiple passes, where each pass handles a subset of the data distribution task. In the first pass, data is distributed from sources to intermediate sinks. In subsequent passes, data is redistributed from intermediate sinks to final sinks. This segmentation reduces the number of mappings each sink must handle in a single pass, transforming the quadratic complexity into a more manageable structure that scales better with the number of sinks.
2Quantity of substance
If the number of sinks is increased to handle larger volumes of data, then the system can accommodate more data, but the number of operations required increases quadratically, resulting in increased overhead and reduced efficiency
Solution Approach 1:
The patent introduces an intermediate dimension by adding intermediate sinks between sources and final sinks. This creates a multi-layered distribution architecture where data flows from sources to intermediate sinks in the first pass, and then from intermediate sinks to final sinks in subsequent passes. This dimensional change allows the system to scale to handle larger volumes of data without the quadratic overhead penalty, as each layer handles a manageable subset of the total data distribution task.
Data Source
AI summary
A system and method for repartitioning data in a distributed network. The method may include executing, by one or more processors, a first pass of a data set from a plurality of first sources to a plurality of first sinks, each first sink collecting data from one or more of the first sources, and executing, by the one or more processors, a second pass of the data set from a plurality of second sources to a plurality of second sinks, each one of the plurality of first sinks corresponding to one of the plurality of second sources, and each second sink collecting data from one or more of the second sources. Executing the first and second passes causes the data set to be repartitioned such that one or more second sinks collect data that originated from two or more of the first sources.