Self-Correcting Complex Event Processing System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing complex event processing systems face challenges in efficiently handling erroneous data without losing crucial information, as they often require complex adaptations and rely on event processing agents to manage uncertainty, leading to reduced accuracy and scalability.
Innovation Solution
A self-correcting complex event processing system with a decoupled error correction component that detects and corrects errors in input events before they reach event processing agents, using a configuration file to define expected behavior and error correction methods, allowing for efficient and scalable error handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If event processing agents directly process raw event streams without error correction, then processing speed is maintained, but data accuracy and reliability deteriorate due to erroneous events
Solution Approach 1:
The system segments the event processing function into two independent components: an error correction component that filters and corrects erroneous events, and event processing agents that handle valid events. This segmentation allows each component to specialize, improving overall reliability without significantly impacting processing speed.
Solution Approach 2:
The error correction component performs preliminary action by detecting and correcting erroneous events before they reach the event processing agents. This preliminary filtering prevents bad data from entering the main processing pipeline, ensuring high reliability while maintaining processing throughput.
2Reliability
If complex error handling mechanisms are implemented within event processing agents, then error detection capability is improved, but system complexity increases
Solution Approach 1:
The error handling functionality is extracted from the event processing agents and placed into a separate error correction component. This extraction simplifies the event processing agents while maintaining comprehensive error detection capabilities in the dedicated correction component.
Solution Approach 2:
The error correction component acts as an intermediary between the event sources and the event processing agents. It mediates the data flow by filtering and correcting events, thereby improving error detection capability without increasing the complexity of the core processing agents.
3Reliability
If all erroneous events are discarded without correction, then data quality is improved, but information loss increases due to potential valid events being removed
Solution Approach 1:
Instead of simply discarding events based on fixed thresholds, the error correction component uses configurable parameters and multiple detection strategies to dynamically assess event validity. This allows for adjustable data quality thresholds that can be tuned to minimize information loss while maintaining high data quality.
Solution Approach 2:
The error correction mechanism is dynamic and adaptive, allowing configuration of correction strategies based on the specific characteristics of the event stream. This dynamic approach enables the system to distinguish between truly erroneous events and valid events with unusual characteristics, reducing information loss while improving data quality.
Data Source
Figure 1~2
Figure 3
AI summary
The present invention relates to a complex event processing, CEP, system (1), comprising an error correction component (20) adapted for receiving a stream of events comprising at least one event from at least one event source (10), wherein the error correction component (20) is adapted for detecting at least one error in the at least one event, and wherein the error correction component (20) is adapted for emitting a corrected stream of events comprising at least one event, which can then be processed by at least one event processing application (40).