Event Processing System Duplicate Detection Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Event processing systems face issues with event duplication due to failures, leading to user experience deterioration and resource wastage, especially during failovers when the backup data center loses track of processed events.
Innovation Solution
Implementing an event tracking system that identifies and blocks duplicate events by comparing event identifiers and information, allowing only permitted events to be processed once, and ensuring seamless failover by replicating event data from the main data center to the backup data center.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the backup data center processes events during failover, then event processing continuity is maintained, but duplicate events may be processed causing user experience deterioration and resource wastage
Solution Approach 1:
The system performs preliminary actions by maintaining an event tracking log that records all processed events before failover occurs. When failover happens, the backup data center uses this pre-established tracking information to identify and filter duplicate events, preventing harmful duplicate processing while maintaining continuity.
Solution Approach 2:
The event tracking system provides feedback by continuously monitoring and recording processed events. During failover, this feedback mechanism allows the backup data center to compare incoming events against the tracking log, identifying duplicates and preventing their processing, thus resolving the contradiction between continuity and duplicate prevention.
2Productivity
If all events are processed without duplication checks, then event processing efficiency is maintained, but events may be processed multiple times causing business impact and resource wastage
Solution Approach 1:
The system extracts only the essential tracking information (event identifiers and processing status) from the full event data into a separate tracking log. This extraction allows for efficient duplicate detection without processing or storing complete event details, maintaining processing efficiency while preventing resource wastage from duplicate handling.
Solution Approach 2:
Instead of duplicating and processing complete event objects, the system creates lightweight copies of event identifiers and tracking metadata. These minimal copies enable rapid duplicate detection with negligible overhead, preserving event processing efficiency while avoiding the resource costs of full event duplication.
3Loss of energy
If duplicate event processing is prevented with strict checking, then resource efficiency is improved, but event processing complexity increases
Solution Approach 1:
The event processing system is segmented into distinct components: event reception, tracking log management, duplicate detection module, and event processing module. This segmentation isolates the complexity of duplicate prevention into a dedicated module, making the overall system more manageable while maintaining resource efficiency through specialized handling.
Solution Approach 2:
An event tracking log serves as an intermediary between event reception and event processing. This mediator component handles the complex task of duplicate detection by comparing incoming events against recorded events, simplifying the main processing logic while ensuring resource efficiency through accurate duplicate prevention.
4Ease of manufacture
If the backup data center loses track of processed events during failover, then failover simplicity is maintained, but duplicate events cannot be identified and processed
Solution Approach 1:
The event tracking functionality is merged into the failover mechanism itself. The tracking log is maintained as part of the data center's state and is transferred or replicated along with other critical data during failover. This merging ensures that event tracking information is preserved during failover without adding separate complex systems.
Solution Approach 2:
The event tracking system serves multiple functions: it tracks processed events for duplicate prevention, maintains data center state for failover, and provides audit information. This multi-functionality allows the same mechanism to support both failover simplicity and event tracking without requiring separate dedicated systems.
Data Source
AI summary
An event processing system may identify duplicate events and prevent one or more of the duplicate events from being processed. The event processing system may determine the duplicate events, which are not permitted to be processed more than once, and allow these duplicate events to be processed exactly once. The event processing system may allow events, which are permitted to be processed more than once, to be processed without determining whether the events are duplicate events. If a main data center of the event processing system fails, a backup data center may take over instantly event processing from the main data center and may prevent events that have been processed by the main data center from being processed. Multiple data centers may work together and process events in an active-active mode by offering the exactly once processing as well as removing any duplicate events from upstream systems.