Streaming Pipeline Slow Instance Mitigation via Watermark Latency Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale streaming pipelines in shared cloud environments face challenges such as resource contention and unavailability of external dependencies, leading to performance deterioration and increased job latencies, with existing approaches either reacting too slowly or overwhelming the resource manager with unnecessary resource requests.
Innovation Solution
A system and method for identifying slow process instance candidates in streaming applications, using a watermark determiner to determine relative latency values and a slow process instance candidate identifier to detect and mitigate slow instances, while applying filters to reduce false positives and optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If an aggressive approach is used to detect and react to slow instances, then responsiveness to environmental changes improves, but the resource manager is overwhelmed with unnecessary resource requests
Solution Approach 1:
The system applies partial action by using filters to process only a subset of slow instance candidates. The first filter removes candidates that are downstream of already-mitigated slow instances, and the second filter removes candidates based on stabilization time criteria. This selective filtering reduces the number of resource requests while maintaining effective detection of genuine slow instances.
Solution Approach 2:
The system introduces intermediary filtering mechanisms between the aggressive detection of slow instances and the resource manager. The filters act as mediators that process candidate instances and selectively pass only the most relevant ones for mitigation, preventing overwhelming the resource manager with unnecessary requests while maintaining aggressive detection capabilities.
2Quantity of substance
If a conservative approach is used to acquire additional resources, then resource manager load is reduced, but the system reacts too slowly to environmental changes
Solution Approach 1:
The system performs preliminary action by pre-processing and filtering slow instance candidates before submitting them for resource allocation. The filtering operations identify and eliminate false positives in advance, so that when resource requests are made, they are more likely to be legitimate and urgent, reducing both the number of requests and the time needed for decision-making.
Solution Approach 2:
The system applies dynamic filtering criteria that adapt to the current state of the streaming pipeline. The stabilization time filter and the filter based on downstream dependencies create a dynamic mechanism that adjusts which instances are flagged for mitigation, allowing the system to respond appropriately to changing environmental conditions without overwhelming the resource manager.
3Loss of time
If all slow process instance candidates are mitigated, then tail latency improves, but the resource manager is overwhelmed with unnecessary resource requests
Solution Approach 1:
The system extracts and removes false positive candidates from the set of slow instance candidates through filtering. The first filter extracts candidates that are downstream of already-mitigated instances, and the second filter extracts candidates that don't meet stabilization time criteria. This leaves only the genuine slow instances that actually need mitigation, improving tail latency while reducing unnecessary resource requests.
Solution Approach 2:
The system applies different filtering criteria to different candidates based on their local characteristics. Candidates are evaluated individually against filters that consider their position in the pipeline, their downstream dependencies, and their stabilization behavior. This localized quality assessment ensures that only candidates with actual mitigation needs are processed further.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
A system is described herein for mitigating slow process instances in a streaming application. The system includes a slow process instance candidate identifier configured to identify, based on a relative watermark latency, a set of slow process instance candidates from among a plurality of process instances that comprise the streaming application. The system further includes a set of filters configured to remove false positives from the set of slow process instance candidates. The filters account for window operations performed by the process instances as well as stabilization time needed for downstream process instances to stabilize after a slow upstream process instance is mitigated by a mitigation implementer, which may also be included in the system.