Network Schedule for Data Re-partitioning in Distributed Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed multi-node systems, data re-partitioning for efficient processing is hindered by significant processing overhead due to data exchange between nodes, which becomes increasingly complex as the number of nodes grows, despite the use of high-speed interconnects like InfiniBand, where re-partitioning can comprise 50-60% of overall query execution time.
Innovation Solution
A greedy selection algorithm is used to determine a network schedule that prioritizes sender nodes with the largest number of data blocks to send and receiver nodes with the largest number of data blocks to receive, optimizing data re-partitioning by minimizing computational overhead and scaling to large multi-node clusters with a time complexity of O(T * N^2).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data re-partitioning is performed in a distributed multi-node system, then data processing efficiency is improved, but processing overhead increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and caching the optimal assignment of data blocks to nodes before actual data re-partitioning occurs. The system calculates the optimal mapping in advance based on data locality requirements and node capabilities, storing this assignment information for reuse during subsequent re-partitioning operations. This eliminates the need to re-calculate optimal assignments during time-critical data exchange phases, significantly reducing processing overhead while maintaining high data processing efficiency
Solution Approach 2:
The patent segments the data re-partitioning process into distinct phases: computation phase for determining optimal assignments, caching phase for storing assignment information, and execution phase for actual data exchange. By segmenting the overall process, the system can perform computationally intensive optimization tasks during idle periods and execute only lightweight operations during active data processing, thereby reducing the time penalty associated with re-partitioning while preserving productivity benefits
2Productivity
If the number of nodes in the distributed system increases, then processing capacity is improved, but re-partitioning complexity increases
Solution Approach 1:
The patent implements self-service by enabling each node to independently determine its optimal data block assignments using locally cached assignment information and local data characteristics. Instead of requiring centralized coordination that scales poorly with node count, each node autonomously retrieves pre-computed assignment rules and applies them to its local data blocks. This self-service approach allows the system to scale to large numbers of nodes without proportionally increasing re-partitioning complexity, as each node operates independently using standardized cached guidance
Solution Approach 2:
The patent changes the parameter representation from detailed node-specific re-partitioning plans to compact assignment rule templates that describe optimal data-block-to-node mappings in terms of data characteristics and node capabilities. By parameterizing the assignment criteria in a generalized form that can be cached and reused, the system reduces the complexity of generating re-partitioning schedules as node count increases. The cached parameters serve as reusable templates that automatically adapt to different numbers of nodes without requiring complete re-computation
3Productivity
If re-partitioning is performed frequently to optimize query execution, then query performance is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent applies preliminary action by pre-computing optimal data block assignments and caching these assignments before actual query execution. When queries require re-partitioning, the system retrieves pre-computed assignments from cache rather than performing new optimization calculations, enabling frequent re-partitioning operations with minimal computational overhead and reduced network bandwidth consumption for control message exchange
Solution Approach 2:
The patent uses copying by creating and storing cached copies of optimal assignment information that can be reused across multiple query executions. Instead of re-transmitting or re-computing assignment data for each query, the system copies and reuses the cached assignment templates, significantly reducing the network bandwidth required for coordination during frequent re-partitioning operations while maintaining optimal query performance
Data Source
Figure 1A
Figure 1B
Figure 1C~1F
AI summary
An approach for efficiently re-partitioning data using scheduled network communication are provided. Given repartitioning data defining the data blocks to be sent amongst a plurality of server nodes, a corresponding network schedule is determined to send the data blocks in a coordinated manner. The network schedule is divided into time slots, wherein each of the plurality of server nodes can send up to one data block and receive up to one data block in each time slot. By using a greedy selection algorithm that prioritizes by largest senders and largest receivers, a near optimal schedule can be determined even in the presence of heavy skew. The greedy selection algorithm can be implemented with a 0(T * N∧2) time complexity, enabling scaling to large multi-node clusters with many server nodes. The network schedule is of particular interest for database execution plans requiring re-partitioning on operators with different keys.