Distributed Map Workflow Parallelization for Large Data Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing workflow systems face limitations in processing large data sets due to data size restrictions, concurrent worker limitations, and logging event restrictions, which hinder efficient data processing and throughput.
Innovation Solution
The workflow system employs a distributed map state that dynamically parallelizes processing by creating as many parallel workflows as there are items in a collection, supports user-defined batching, and treats each child execution as a separate workflow to overcome data size and logging limits, allowing for more concurrent iterations and improved logging capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data size limitations are enforced in workflow steps, then system stability is maintained, but data processing capability is restricted
Solution Approach 1:
The patent segments large data sets into smaller batches that can be processed individually through workflow steps. The distributed map state divides the collection of items into multiple batches, each within the data size limit, allowing processing of large data sets by breaking them into manageable segments that traverse the workflow repeatedly.
Solution Approach 2:
The patent introduces a new dimension of parallel execution by creating multiple child workflow executions that run concurrently. Instead of processing one large data set sequentially, the system creates multiple parallel workflow instances, each handling a portion of the data, thereby overcoming the data size limitation through dimensional expansion in the execution space.
2Use of energy by moving object
If the number of concurrent workers is limited, then system resource consumption is controlled, but data processing throughput is reduced
Solution Approach 1:
The patent implements dynamic parallelization where the number of concurrent workers is not fixed but adapts based on the size of the data collection. The system dynamically creates child workflow executions up to a maximum threshold, allowing the concurrency level to flex according to workload demands rather than being constrained by a static limit.
Solution Approach 2:
The patent changes the parameter of worker concurrency from a fixed value to a dynamic value that can scale with data collection size. By adjusting the number of parallel workflow executions based on the input data size, the system optimizes resource utilization while maintaining high throughput for large data sets.
3Power
If logging events are restricted per workflow execution, then logging system performance is maintained, but monitoring capability for large data sets is insufficient
Solution Approach 1:
The patent segments the logging burden by creating multiple child workflow executions, each with its own logging capacity. Instead of one workflow trying to log all events, multiple workflows distribute the logging load, with each workflow logging events for its specific batch of processed items, thereby maintaining logging performance while capturing comprehensive monitoring data.
4Device complexity
If fixed threshold limits are applied to parallel workflows, then system control is simplified, but adaptability to different data sizes is reduced
Solution Approach 1:
The patent transitions from static threshold limits to dynamic thresholds that adapt to the size of the input data collection. The system calculates the maximum number of parallel workflows based on the collection size and a configured threshold, allowing the parallelization level to automatically adjust to different data sizes while maintaining simplified control through the threshold parameter.
Data Source
AI summary
System and methods are provided for dynamically parallelized large data set processing. A workflow includes a distributed map state. A set of steps in the distributed map state are run for each item in a collection of items or for each batch. Execution of the distributed map state is dynamically parallelized such that the system creates as many parallel workflows as there are items in the collection at run time (or batches). The items in the collection are dynamically determined by the workflow system, as defined by a resource in the distributed map state.