Stream Processing Runtime Adaptation via Event-Driven Orchestration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream processing applications lack constructs for runtime adaptation to changes in conditions, such as high input data rates, due to the declarative nature of existing languages, which focuses on data processing logic rather than adapting to runtime changes.
Innovation Solution
A method for providing stream processing with runtime adaptation by registering events associated with stream processing applications, monitoring for event occurrences, receiving event notifications, and executing adaptations based on runtime metrics, utilizing an orchestrator that communicates with the runtime infrastructure and stream processing application to implement event detection and actuation policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If stream processing applications use declarative languages to focus on data processing logic, then data processing logic expression is improved, but runtime adaptation capability deteriorates
Solution Approach 1:
The system segments stream processing applications into event definitions, event handlers, and processing logic. Event definitions capture runtime conditions (e.g., high data rates, resource utilization thresholds) separately from processing logic, allowing independent adaptation without rewriting core processing code. This segmentation enables declarative logic to remain unchanged while adding adaptive capabilities through separate event handling components.
Solution Approach 2:
Event handlers serve as intermediaries between runtime conditions and stream processing applications. When runtime metrics indicate a condition change (such as overload or resource availability), event handlers mediate by triggering appropriate responses (load shedding, scaling, reconfiguration) without requiring the processing logic itself to be imperative or adaptive. This intermediary layer preserves the simplicity of declarative languages while enabling runtime adaptation.
2Duration of action of stationary object
If stream processing applications continuously run until explicitly cancelled, then application execution continuity is improved, but ability to respond to runtime changes deteriorates
Solution Approach 1:
The system implements continuous feedback loops by monitoring runtime metrics (data rates, resource utilization, queue depths) during application execution. Event handlers receive feedback about current system state and trigger adaptations when thresholds are exceeded. This feedback mechanism allows continuously running applications to dynamically respond to runtime conditions through load shedding, scaling, or reconfiguration events, resolving the contradiction between continuous execution and adaptive response.
3Reliability
If stream processing applications apply load shedding policies during overload, then quality of service is improved, but system complexity increases
Solution Approach 1:
The system dynamically adjusts processing behavior based on runtime conditions through event-driven load shedding. When monitoring detects overload conditions (high data rates, resource exhaustion), event handlers dynamically apply load shedding policies by dropping or deferring tuples. When conditions improve, normal processing resumes. This dynamic approach maintains quality of service guarantees during overload without requiring permanent complex infrastructure, as the complexity is activated only when needed.
Data Source
AI summary
Embodiments of the disclosure include a method for providing stream processing with runtime adaptation includes registering one or more events, wherein each of the events is associated with a stream processing application. The method also includes monitoring, by a processor, for an occurrence of the one or more events associated with the stream processing application, wherein each of the one or more events is associated with one or more runtime metrics. The method further includes receiving an event notification, wherein the event notification includes event identification and an event context and executing an adaptation of the stream processing application.