Parallel Ordered Data Stream Processing with Order Keys
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data parallel processing, the ordering of input data elements can be lost when partitions are non-contiguous, leading to incorrect ordering of output data streams if not explicitly tracked.
Innovation Solution
Assigning ordinal order indices to input data elements during partitioning and maintaining them during parallel execution to ensure correct ordering of output data streams, using a data structure that stores elements along with their order keys to determine the correct order.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data parallel processing is used to improve processing speed, then productivity increases, but the ordering of output data stream deteriorates
Solution Approach 1:
The patent applies preliminary action by assigning order keys to data elements before partitioning the data stream for parallel processing. This pre-assignment of ordering information ensures that when data is distributed across multiple workers and processed concurrently, the original ordering can be reconstructed by sorting on these pre-assigned keys, thus maintaining ordering accuracy while achieving parallel processing speedup.
2Productivity
If non-contiguous partitioning is used to distribute data across workers, then productivity improves through better load balancing, but the ordering information is lost
Solution Approach 1:
The patent uses order keys as an intermediary mechanism that bridges the gap between non-contiguous data partitioning and ordering preservation. These order keys are attached to data elements and serve as metadata that carries ordering information through the parallel processing pipeline, allowing workers to process data in any order while enabling correct reordering at the output stage.
3Manufacturing precision
If sequential processing is used to maintain data ordering, then ordering accuracy is preserved, but productivity decreases
Solution Approach 1:
The patent applies segmentation by dividing the data stream into partitions that can be processed independently in parallel by multiple workers. Each partition is processed concurrently, and the use of order keys enables the segmented results to be reassembled in the correct order, thus achieving both parallel processing speedup and ordering accuracy.
Data Source
AI summary
A method of parallel processing an ordered input data stream that includes a plurality of input data elements and a corresponding plurality of order keys for indicating an ordering of the input data elements, with each order key associated with one of the input data elements, includes processing the input data stream in a parallel manner with a plurality of worker units, thereby generating a plurality of sets of output data elements. The plurality of sets of output data elements is stored in a plurality of buffers, with each buffer associated with one of the worker units. An ordered output data stream is output while the input data stream is being processed by outputting selected output data elements from the buffers in an order that is based on the order keys.