Repartitioning Optimizer for Distributed Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In massively distributed computation, data repartitioning introduces significant delays due to network communication and inefficient input/output operations, particularly with traditional magnetic storage media.
Innovation Solution
A repartitioning optimizer identifies and selects optimal strategies based on processing, local I/O, and network transfer utilization, and assigns repartitioning indices to data elements for sequential storage, reducing randomized input/output operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is redistributed among computing devices for subsequent tasks, then data accessibility for new tasks is improved, but network communication time increases
Solution Approach 1:
The system performs preliminary partitioning of data at each computing device before redistribution is needed. By pre-organizing data into partitions that can be efficiently transferred, the system reduces the time required for actual data redistribution when subsequent tasks require different data subsets.
Solution Approach 2:
The patent divides data into discrete partitions at each computing device, allowing selective transfer of only necessary partitions rather than redistributing entire datasets. This segmentation enables efficient data redistribution by transferring only the required portions to relevant computing devices.
2Productivity
If data is partitioned locally at each computing device, then processing efficiency is improved, but input/output operations become randomized and inefficient
Solution Approach 1:
The system performs preliminary sorting of data partitions at each computing device before transfer. By pre-organizing data in sorted order, the system converts randomized I/O operations into sequential I/O operations, which are significantly more efficient for magnetic storage media and reduces the time required for data partitioning and transfer.
Solution Approach 2:
The patent changes the organizational parameter of data from unsorted/random to sorted/sequential. This parameter change transforms the nature of I/O operations from inefficient randomized access to efficient sequential access, directly addressing the performance issue with traditional magnetic storage media.
3Adaptability or versatility
If large amounts of data are transferred over the network, then data redistribution is achieved, but transfer time becomes substantial compared to processing time
Solution Approach 1:
The system extracts and transfers only the necessary data partitions required for subsequent tasks, rather than transferring entire datasets. This selective extraction minimizes the volume of data transferred over the network while still achieving the required data redistribution for task execution.
Solution Approach 2:
The system performs preliminary identification of which data partitions are needed for subsequent tasks before initiating transfer. This preliminary action allows the system to prepare transfer lists and optimize network utilization by transferring only essential data, reducing overall transfer time.
Data Source
AI summary
A repartitioning optimizer identifies alternative repartitioning strategies and selects optimal ones, accounting for network transfer utilization and partition sizes in addition to traditional metrics. If prior partitioning was hash-based, the repartitioning optimizer can determine whether a hash-based repartitioning can result in not every computing device providing data to every other computing device. If prior partitioning was range-based, the repartitioning optimizer can determine whether a range-based repartitioning can generate similarly sized output partitions while aligning input and output partition boundaries, increasing the number of computing devices that do not provide data to every other computing device. Individual computing devices, as they are performing a repartitioning, assign a repartitioning index to each individual data element, which represents the computing device to which such a data element is destined. The indexed data is sorted by such repartitioning indices, thereby grouping together all like data, and then stored in a sequential manner.