Stream Processor Deduplication for Exactly-Once Delivery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data pipeline architectures, such as Lambda and Kappa, face inefficiencies in processing data in real-time while maintaining integrity and complex operations, leading to hardware duplication, software complexity, and increased latency, which affects data certification and reporting accuracy.
Innovation Solution
A new data processing system utilizing a stream processing engine with deduplication and audit processes to identify and replay missing events, ensuring data integrity and reducing the need for batch processing, thereby eliminating hardware and software duplication and enabling real-time certification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If Lambda pipeline architecture is used with both batch and stream processors, then real-time processing speed is improved, but hardware duplication and system complexity increase
Solution Approach 1:
The patent merges batch and stream processing into a unified stream processing architecture. The stream processor handles both real-time streaming data and batch reprocessing tasks through a single codebase, eliminating the need for separate batch processing infrastructure while maintaining both real-time speed and batch processing capabilities.
Solution Approach 2:
The stream processor is designed to perform multiple functions: real-time event processing, batch reprocessing, and exactly-once delivery guarantees. This universal processor eliminates the need for specialized batch processing hardware by making the stream processor multi-functional.
2Device complexity
If Kappa pipeline architecture is used with stream processor only, then hardware duplication is reduced, but processing speed decreases due to frequent re-running
Solution Approach 1:
The system performs preliminary deduplication and processing validation during the initial stream processing pass. This preliminary action ensures that when batch reprocessing is needed, the stream processor can efficiently reprocess data without redundant operations, maintaining speed while using a single processor.
Solution Approach 2:
The stream processor maintains continuous operation by seamlessly switching between real-time processing and batch reprocessing modes. The single processor continuously handles streaming data while periodically reprocessing historical data, eliminating idle periods and maintaining high utilization without hardware duplication.
3Speed
If stream processing is used for real-time data processing, then processing speed is improved, but data integrity and certification accuracy deteriorate
Solution Approach 1:
The system implements feedback through an audit process that continuously monitors stream processing output for data integrity. The auditor compares processed events against expected results and triggers batch reprocessing when discrepancies are detected, ensuring data integrity while maintaining real-time processing speed through selective validation.
Solution Approach 2:
Deduplication is performed as a preliminary action during stream processing, with events being de-duplicated before output. This preliminary deduplication prevents duplicate events from propagating through the system, maintaining data integrity without requiring post-processing correction.
4Reliability
If batch processing is used to ensure data certification accuracy, then data integrity is improved, but processing time and latency increase
Solution Approach 1:
Instead of performing full batch processing for all data, the system performs partial batch reprocessing only for events that require integrity verification. The stream processor handles the majority of data in real-time, while batch processing is applied selectively to ensure data certification accuracy only when needed, minimizing time loss.
Data Source
AI summary
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising processing a plurality of events using a stream processor and executing a deduplication process on the plurality of events using the stream processor. The plurality of events is outputted to a streaming queue and a close of books (COB) of a data transport is detected. Then, an audit process is initiated in response to detecting the COB signal, the audit process comprising comparing a set of raw events to a set of events in the streaming queue to identify a set of missing events, and replaying a set of missing events through the stream processor.