Dynamic Dataset Chunk Splitting for Distributed Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed processing systems for large datasets require upfront splitting of data into chunks, which limits flexibility and efficiency as the number of workers can change dynamically, and often necessitate communication with a central dispatcher or storage, leading to suboptimal resource utilization.
Innovation Solution
A method where workers can dynamically split large datasets into chunks without contacting the storage, using integer intervals and arithmetic operations to assign and redistribute data records among available workers, ensuring no overlap or gaps, and allowing for efficient redistribution of remaining work as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the dataset is split into chunks upfront by a central dispatcher or storage, then the data distribution is initially established, but the system cannot adapt dynamically when the number of workers changes and requires continuous communication with the central dispatcher
Solution Approach 1:
The patent implements dynamic chunk splitting where workers can request splits of other workers' chunks during processing. The system transitions from static upfront splitting to dynamic on-demand splitting, allowing the data distribution to adapt as workers are added or removed without requiring centralized recoordination.
Solution Approach 2:
Workers autonomously perform chunk splitting operations without contacting the central dispatcher or storage. When a worker needs additional work, it can request a split of an existing chunk from another worker, and the splitting is performed locally using arithmetic operations on key ranges, eliminating the need for centralized coordination.
2Productivity
If the dataset is physically split up-front into a fixed number of chunks, then data distribution is simplified, but the processing capacity utilization becomes suboptimal when worker availability changes
Solution Approach 1:
The system allows chunks to be dynamically split during processing based on current worker availability. When workers are added to the system, existing chunks can be split to create new work assignments, ensuring that processing capacity utilization is optimized regardless of when workers join or leave.
Solution Approach 2:
The patent enables further segmentation of existing chunks during processing. Instead of having a fixed chunk structure determined upfront, chunks can be divided into smaller sub-chunks on-demand, allowing the system to create more granular work assignments as workers become available.
3Loss of time
If workers communicate with the central dispatcher during processing, then task assignment is coordinated, but communication overhead and processing delays increase
Solution Approach 1:
Workers perform chunk splitting operations autonomously without contacting the central dispatcher. The splitting logic is implemented locally using arithmetic operations on key ranges, eliminating communication overhead and allowing workers to independently adapt their work assignments in real-time.
Solution Approach 2:
The chunk splitting functionality is extracted from the central dispatcher and implemented directly in the workers. This removes the communication dependency for splitting operations, allowing workers to independently manage their own chunk divisions without involving the central coordination service.
4Adaptability or versatility
If the dataset is enumerated by a central dispatcher, then data distribution is controlled, but the system requires continuous contact with the dispatcher and cannot operate independently
Solution Approach 1:
Workers independently perform chunk splitting using local computation based on key range arithmetic. The system eliminates the need for central enumeration by enabling workers to autonomously determine chunk boundaries through mathematical operations on key values, allowing independent operation without dispatcher contact.
Solution Approach 2:
The patent replaces the mechanical central enumeration process with arithmetic operations on key ranges. Instead of the central dispatcher physically enumerating and distributing data, workers use mathematical calculations on key values to determine chunk boundaries and split chunks locally.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
When a new worker requests access to a dataset, the largest chunk of the dataset is identified and split into two new chunks by the worker having the chunk assigned to it. The chunk is split in such a manner that both workers have enough un-processed data records, and collisions among the workers processing the data records are avoided. Finding the split point may be an iterative process.