Event Stream Processing Model Generation with Parallel Data Enrichment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying analytic tools and models in edge devices and the cloud to handle large-scale, geographically distributed, and highly dynamic sensors is challenging, particularly in applications like predictive maintenance for vehicles, where real-time data processing and immediate reactions are required.
Innovation Solution
An event stream processing (ESP) system that processes events by reading an ESP model, extracting new measurement values, updating an in-memory data store, and using data enrichment windows to generate enriched data values, which are then output when an output update time is reached, enabling real-time analytics and alerts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If analysts and software developers manually design and develop source code to implement ESP models, then the analytics problems can be solved with custom tailored solutions, but significant development and testing time is required
Solution Approach 1:
The system performs preliminary actions by automatically generating ESP model source code from high-level analytics specifications before manual development would begin. The code generation engine pre-compiles templates and parameters into complete, ready-to-deploy source code, eliminating the need for analysts to manually write and test code from scratch.
Solution Approach 2:
The system uses code templates that can be copied and adapted for different analytics problems. Instead of manually creating unique code for each analytics task, the system copies proven, pre-tested code templates and automatically instantiates them with problem-specific parameters, ensuring reliability while reducing development time.
2Speed
If ESP models process millions of events per second with sub-millisecond response times, then real-time analytics capability is achieved, but system complexity increases
Solution Approach 1:
The ESP system is segmented into distinct, modular components: event ingestion modules, windowing operators, aggregation functions, and output modules. Each component handles a specific aspect of processing, allowing the system to achieve high throughput through specialized optimization while managing complexity through clear separation of concerns.
Solution Approach 2:
The system employs universal data structures and processing frameworks that can handle multiple types of events and analytics operations through a common interface. The windowing and aggregation mechanisms are designed to work with any event type, reducing the need for separate complex handling logic for different scenarios.
3Measurement precision
If data enrichment windows compute values from multiple windows using gate windows, then data accuracy and completeness are improved, but processing overhead increases
Solution Approach 1:
The gate window mechanism uses periodic triggering based on event counts or time intervals to initiate computations from multiple input windows. Instead of continuously computing, the system waits for periodic triggers (such as receiving a threshold number of events or a time-based interval), reducing processing overhead while maintaining data accuracy through regular updates.
Solution Approach 2:
Input windows perform preliminary computations and data preparation in advance, organizing data into ready-to-use formats before the gate window needs it. This preliminary action reduces the computational burden on the gate window, allowing it to focus on combining results from multiple sources with minimal additional processing overhead.
Data Source
AI summary
An event stream processing (ESP) model is read that describes computational processes. (A) An event block object is received. (B) A new measurement value, a timestamp value, and a sensor identifier are extracted. (C) An in-memory data store is updated with the new measurement value, the timestamp value, and the sensor identifier. (A) through (C) are repeated until an output update time is reached. When the output update time is reached, data stored in the in-memory data store is processed and updated using data enrichment windows to define enriched data values that are output. The data enrichment windows include a gate window before each window that uses values computed by more than one window. The gate window sends a trigger to a next window when each value of the more than one window has been computed. The enrichment windows are included in the ESP model.