Data Merging System for Low-Latency Real-Time and Bulk Data Reconciliation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently merging real-time data and bulk data, particularly in serving large datasets exceeding 20 billion records to tens of millions of users at low latency while maintaining correctness and consistency, with no practical scheme available to combine these data types effectively.
Innovation Solution
The system employs a predictive activity queue and a 'Warmr' component that observes and scans HBase to project entities to a low-latency serving tier, using a 'real time trumps backfill' algorithm to prioritize real-time data, minimizing HBase scan IO, and orchestrating a cleanup phase to manage data consistency, allowing for surgical updates and whole-dataset updates while maintaining high consistency and correctness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If real-time data and bulk data are merged using existing distributed database systems like HBase, then data serving capability is improved, but data consistency and correctness deteriorate due to lack of merging schemes
Solution Approach 1:
The patent segments data into two distinct types: real-time data and bulk data. This segmentation allows each data type to be handled with appropriate merging strategies, preventing consistency issues that arise from treating all data uniformly. The system maintains separate data streams and applies specific reconciliation rules for each type.
Solution Approach 2:
The patent introduces an intermediary merging layer that sits between the distributed database and the serving system. This intermediary component reconciles real-time and bulk data before serving, ensuring consistency while maintaining the high throughput capabilities of the underlying database system.
2Reliability
If full update of data is performed to maintain consistency, then data correctness is improved, but serving latency increases due to system load
Solution Approach 1:
Instead of performing full data updates, the patent applies partial updates only to the specific portions of data that have changed. The system identifies and reconciles only the affected data segments, significantly reducing the computational overhead and maintaining low serving latency while ensuring correctness.
Solution Approach 2:
The system performs preliminary identification of changed data segments before executing updates. By pre-processing and identifying only the necessary data portions that require reconciliation, the system avoids unnecessary full-scans and updates, thereby maintaining both correctness and low latency.
3Quantity of substance
If large datasets exceeding 20 billion records are served to tens of millions of users, then data completeness is improved, but system resource requirements increase
Solution Approach 1:
The patent merges real-time and bulk data processing paths into a unified serving architecture. This consolidation allows the system to handle 20+ billion records efficiently by reusing infrastructure components and avoiding duplicate processing, thereby reducing overall hardware requirements while maintaining data completeness.
Data Source
AI summary
The present teaching relates to methods, systems, and programming for reconciling or merging real time data with bulk data. In one example, a first attribute value is obtained. The first attribute value characterizes an aspect of a data item. A second attribute value related to the first attribute value is obtained. The second attribute value characterizes the aspect of the data item. A scheme is selected to be used to determine a third attribute value to characterize the data item. The third attribute value is determined in accordance with the selected scheme as well as the first and second attribute values.