Stream Processing Framework for Strong Ordering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing systems face inefficiencies due to round-robin task distribution strategies, leading to significant latency and resource wastage, especially in environments with varying computational demands and limited resources, making them costly for small or variable workloads.
Innovation Solution
Implementing a combination of concurrent and multiplexed processing schemes within a stream processing framework that adapts to available computational resources by concurrently processing batches when resources are sufficient and multiplexing when resources are limited, while maintaining strong ordering in multi-stage processing of near real-time data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If round-robin task distribution strategy is used, then tasks are evenly distributed between computation resources, but substantial computation time and resources are lost during assignment and re-assignment, introducing significant latency
Solution Approach 1:
The system performs preliminary actions by pre-assigning tasks to computation resources based on predicted availability and task requirements before actual execution. The task assignment module proactively matches tasks with suitable computation resources in advance, reducing the need for frequent re-assignment and minimizing latency during task distribution.
Solution Approach 2:
The system implements dynamic task assignment that adapts to changing computational requirements and resource availability. Instead of static round-robin distribution, the task assignment module continuously monitors resource status and dynamically reassigns tasks to optimize performance while minimizing disruption and latency.
2Ease of operation
If round-robin task distribution strategy is used, then tasks are evenly distributed, but substantial computation resources are lost during assignment and re-assignment, making it not cost effective
Solution Approach 1:
The system performs preliminary matching of tasks with computation resources based on predicted availability and task requirements. This pre-assignment reduces the frequency of re-assignment operations, thereby minimizing the computational overhead and resource wastage associated with task management.
Solution Approach 2:
The system changes the parameters of task assignment by considering multiple factors such as computation resource availability, task priority, and expected execution time. This multi-parameter optimization approach ensures that tasks are assigned to the most suitable resources, reducing unnecessary re-assignment and improving overall resource utilization efficiency.
3Productivity
If concurrent processing is implemented when resources are sufficient, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The system dynamically adjusts the processing mode based on real-time resource availability. When computation resources are sufficient, it switches to concurrent processing to maximize productivity. When resources are limited, it transitions to multiplexed processing. This dynamic adaptation allows the system to achieve high efficiency without permanently increasing system complexity.
Solution Approach 2:
The processing system automatically monitors its own resource status and makes decisions about whether to use concurrent or multiplexed processing without external intervention. This self-service capability simplifies the overall system architecture by embedding the complexity management within the processing module itself.
Data Source
AI summary
The technology disclosed relates to providing strong ordering in multi-stage processing of near real-time (NRT) data streams. In particular, it relates to maintaining current batch-stage information for a batch at a grid-scheduler in communication with a grid-coordinator that controls dispatch of batch-units to the physical threads for a batch-stage. This includes operating a computing grid, and queuing data from the NRT data streams as batches in pipelines for processing over multiple stages in the computing grid. Also included is determining, for a current batch-stage, batch-units pending dispatch, in response to receiving the current batch-stage information; identifying physical threads that processed batch-units for a previous batch-stage on which the current batch-stage depends and have registered pending tasks for the current batch-stage; and dispatching the batch-units for the current batch-stage to the identified physical threads subsequent to complete processing of the batch-units for the previous batch-stage.