Variable Parallelism Layouts for Dataflow Query Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems using dataflow graphs assign uniform parallelism to all nodes, failing to account for varying degrees of parallelism required by different datasets, leading to inefficient data processing.
Innovation Solution
Techniques for automatically determining processing layouts with varying degrees of parallelism for different nodes in a dataflow graph, allowing repartitioning operations to adapt data processing based on the specific requirements of input and output datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If uniform parallelism is assigned to all nodes in a dataflow graph, then the system structure is simple and easy to manage, but data processing efficiency deteriorates when datasets have different parallelism requirements
Solution Approach 1:
The patent applies local quality by assigning different processing layouts with varying degrees of parallelism to different nodes in the dataflow graph based on their specific requirements. Each node can have a customized processing layout that matches the parallelism characteristics of its input and output datasets, rather than applying a uniform parallelism level across all nodes. This resolves the contradiction by allowing the system to maintain simplicity in layout determination for uniform cases while achieving high processing efficiency when diverse parallelism requirements exist.
2Productivity
If variable parallelism is implemented for different nodes, then data processing efficiency is improved, but the complexity of determining processing layouts increases
Solution Approach 1:
The patent applies preliminary action by determining processing layouts for nodes in a systematic sequence: first obtaining layouts for input nodes, then using those to determine layouts for intermediate nodes, and finally determining layouts for output nodes. This staged approach breaks down the complex task of variable parallelism assignment into manageable steps, reducing the overall complexity while maintaining the benefits of optimized parallelism at each stage of the dataflow graph.
3Adaptability or versatility
If repartitioning operations are added to adapt to different dataset requirements, then adaptability is improved, but the number of operations and system complexity increases
Solution Approach 1:
The patent applies the intermediary principle by introducing repartitioning operations as mediator nodes between nodes with different processing layouts. These repartitioning operations serve as intermediaries that transform data between different parallelism configurations, enabling seamless adaptation to varying dataset requirements. The repartitioning nodes act as buffers that reconcile the differences in parallelism between adjacent nodes in the dataflow graph, providing adaptability while managing complexity through a standardized intermediary mechanism.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Techniques for determining processing layouts to computer code portions of a computer program. The techniques include: obtaining computer code that, when executed by at least one computer hardware processor, causes the at least one computer hardware processor to execute a database query, wherein the computer code comprises: a first set of one or more computer code portions each representing a data processing operation for reading in a respective input dataset; a second set of one or more computer code portions each representing a data processing operation for writing out a respective output dataset; and a third set of one or more computer code portions each representing a respective data processing operation; obtaining a first set of one or more processing layouts for one or more code portions in the first set of code portions; obtaining a second set of one or more processing layouts for one or more code portions in the second set of code portions; and determining a processing layout for each code portion in the third set of code portions based on the first set of processing layouts, the second set of processing layouts, and one or more layout determination rules including at least one rule for selecting among processing layouts having different degrees of parallelism.