Real-Time Data Capture System for Batch Processing Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current batch processing systems face challenges in handling high-volume, real-time data processing, particularly in maintaining data quality and handling diverse data types, which can lead to inefficiencies and data loss during processing.
Innovation Solution
A system and method for batch processing that involves receiving and decomposing events into insert, update, and delete operations, determining schemas using fingerprints, tokenizing sensitive information, and storing data in a database with mirroring for redundancy, enabling real-time data capture and processing while ensuring data quality and supporting multiple data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If real-time data processing is implemented, then data processing speed and responsiveness are improved, but data quality and reliability deteriorate due to processing errors and data loss
Solution Approach 1:
The system performs preliminary actions by capturing data changes at the source before processing, using change data capture (CDC) to record all modifications to source data. This preliminary capture ensures data is preserved in its original state, allowing for reliable reconstruction and processing without losing data quality or incurring processing errors.
Solution Approach 2:
The system creates accurate copies of source data through CDC mechanisms, capturing changes and storing them in a change data store. These copies maintain the integrity and quality of original data while enabling parallel processing operations, thus improving productivity without compromising data reliability.
2Adaptability or versatility
If diverse data types are processed, then system versatility is improved, but processing complexity and data quality management worsen
Solution Approach 1:
The system implements a universal processing architecture that handles multiple data types through a common change data capture mechanism. The CDC framework captures changes across different data types (numeric, character, date, etc.) using the same underlying technology, eliminating the need for separate processing paths and reducing overall system complexity while maintaining versatility.
Solution Approach 2:
The system manages diverse data types by treating them as parameter variations rather than fundamentally different structures. By capturing data changes through a unified CDC approach that adapts to different data types through parameter specifications, the system maintains simplicity while handling diversity, avoiding the complexity that would arise from type-specific processing logic.
3Reliability
If batch processing is used, then data quality checks are improved, but processing time and latency worsen
Solution Approach 1:
The system implements continuous change data capture that operates continuously in the background, capturing data changes as they occur without interrupting source systems or requiring batch processing cycles. This continuous operation maintains data quality through ongoing validation while eliminating the time delays associated with periodic batch processing, thereby reducing latency while preserving quality assurance.
Data Source
AI summary
The exemplary embodiments provide real-time data capture and processing which improves data processing performance and speed and facilitate passing of the processed data to various analytical sources, while maintaining superior data quality checks, particularly with respect to data elements associated with multiple data types. The proposed system and process can be used to continuously consume and listen to multiple events while mapping the events to appropriate schemas provided in a separate schema stream. The schema stream is provided once and cached to minimize bandwidth consumed by the transaction stream. The schema information is then further enriched with information from a metadata registry. The event data may then be compressed and aligned in memory tables based on the enriched schema. Once events are decoded and sorted into memory tables in accordance to the identified schema, each memory table can be processed in parallel.