Streaming Data Reconciliation with Independent Reference Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reconciliation techniques struggle to ensure high data quality and completeness in complex data transmission processes, especially in heavily regulated industries, due to the complexity of data filters and transformations.
Innovation Solution
The use of an independent reconciliation data stream facilitated by machine learning and artificial intelligence to monitor and validate streaming data in near real-time, identifying data anomalies and generating alerts when parameters exceed predefined thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional reconciliation techniques are used with multiple data filters and transformations, then regulatory requirements are met, but data quality monitoring becomes difficult and complex
Solution Approach 1:
The patent introduces an independent reconciliation data stream that acts as an intermediary between the original data stream and the monitoring system. This reconciliation stream captures the expected state of data after transformations, allowing monitoring without directly analyzing the complex transformation pipeline. The reconciliation stream serves as a mediator that simplifies quality assurance by providing a reference point for comparison.
Solution Approach 2:
The system implements feedback by comparing events in the original data stream with corresponding events in the reconciliation data stream. This feedback mechanism identifies anomalies by detecting deviations between expected and actual data states, enabling continuous quality monitoring without requiring direct analysis of the complex transformation processes.
2Reliability
If data streams are enhanced with reference data and aggregated into various views, then data completeness is improved, but monitoring and validation becomes more difficult
Solution Approach 1:
The reconciliation data stream serves as an intermediary that maintains the expected state of enhanced and aggregated data. By comparing actual data against this reconciliation stream, the system can monitor data completeness and validity without directly analyzing the complex aggregated views, thus reducing monitoring difficulty while maintaining data completeness.
Solution Approach 2:
The system creates a copy of the data stream in the form of a reconciliation data stream that preserves the expected state of enhanced and aggregated data. This copy allows for simplified monitoring and validation by providing a reference framework against which actual data can be compared, avoiding the complexity of directly analyzing aggregated views.
3Measurement precision
If machine learning models are used for near real-time monitoring, then data quality detection is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary action by pre-processing data into the reconciliation data stream that captures the expected state. This pre-computed reference framework enables faster real-time monitoring by providing a ready-made basis for comparison, reducing the computational burden during actual anomaly detection while maintaining high precision through machine learning models.
Data Source
AI summary
A method for facilitating reconciliation of streaming data by using an independent reconciliation data stream is disclosed. The method includes receiving events from a data stream; accessing a reconciliation data stream that corresponds to the data stream, the reconciliation data stream including reconciliation events that correspond to the events; determining, by using a model, parameters based on the events and the corresponding reconciliation events, the parameters relating to the data stream; comparing, by using the model, the events with the corresponding reconciliation events to identify data anomalies; validating, by using the model, the events based on the corresponding reconciliation events; and aggregating information that relates to the events, the parameters, the data anomalies, and a result of the validating.