Stream Processing Event Certification via Fact COB Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data pipeline architectures, such as Lambda and Kappa, face inefficiencies in processing data in real-time while maintaining data integrity and complexity, requiring duplication of processing logic and hardware, which leads to increased latency and variability in reporting.
Innovation Solution
A new data processing system utilizing a stream processing engine with additional hardware and software to detect and correct inaccuracies during stream processing, allowing for real-time certification of events without the need for batch processing, thereby reducing processing and certification times and eliminating the requirement for duplicate hardware and software resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a batch processor is used to ensure data integrity and perform complex operations, then data processing accuracy is improved, but processing speed deteriorates due to the inability to process data in real-time
Solution Approach 1:
The patent performs a fact COB computation before certifying processed events to proactively identify missing events. This preliminary action ensures data integrity is verified in advance, allowing real-time certification without requiring subsequent batch processing corrections.
Solution Approach 2:
The system implements a feedback mechanism where the fact COB computation results (identifying missing events) are used to determine whether to certify processed events. This feedback loop ensures accuracy by continuously verifying data completeness before final certification, eliminating the need for batch re-processing.
2Speed
If a stream processor is used to achieve real-time processing, then processing speed is improved, but data integrity deteriorates due to potential data loss or duplication
Solution Approach 1:
The system performs a fact COB computation before certification to proactively identify any missing events in the stream processing output. This preliminary verification ensures data integrity is confirmed in real-time, eliminating the need for batch re-processing while maintaining complete data accuracy.
Solution Approach 2:
The patent replaces the mechanical batch re-processing system with a computational verification system. Instead of re-running stream processors to ensure integrity, the system uses a fact COB computation to mathematically verify that all events are present and accounted for, maintaining real-time processing while ensuring data completeness.
3Reliability
If Lambda architecture with both batch and stream processors is used, then both real-time processing and data integrity are improved, but device complexity increases due to duplication of processing logic and hardware
Solution Approach 1:
The patent extracts the data integrity verification function from the batch processing system and implements it as a standalone fact COB computation that works with stream processing output. This extraction eliminates the need for duplicate batch processing logic while maintaining integrity verification, reducing overall system complexity.
Solution Approach 2:
The fact COB computation serves multiple functions: it verifies data completeness, identifies missing events, and enables certification decisions. This multi-functional component replaces the need for separate batch processing logic, allowing a single stream processing system to achieve both real-time performance and data integrity.
4Device complexity
If Kappa architecture with stream processor is used, then hardware complexity is reduced by eliminating batch processor, but processing speed deteriorates due to frequent re-running of event streams
Solution Approach 1:
The system performs a fact COB computation before certification to identify missing events in advance. This preliminary action eliminates the need for time-consuming re-running of event streams, as any missing events are detected and handled before the certification decision is made, maintaining low latency.
Solution Approach 2:
The stream processing system serves its own integrity verification needs through the fact COB computation. Instead of requiring external batch processing or re-running mechanisms, the system self-verify data completeness through computational comparison of processed events against expected event counts, eliminating time-wasting re-processing cycles.
Data Source
AI summary
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising detecting a real close of books (COB) of a data transport, the real COB associated with a set of raw events transmitted over the data transport, flushing a stream processor in response to detecting the real COB, and retrieving a set of processed events from a distributed file system after the flushing is complete. A fact COB computation is then performed on the set of processed events and the set of raw events, the fact COB computation outputting a number of missing events, each missing event representing a raw event that is not present in the set of processed events. The processed events are then certified upon determining that the number of missing events is below a threshold.