Aggregate Data Verification Engine for Disparate Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for verifying data integrity in large, disparate databases are inefficient and prone to errors due to the need for extensive comparison of entire data structures, making it difficult to detect and correct discrepancies in replicated data across multiple databases.
Innovation Solution
An evaluation engine is used to generate and compare aggregates from disparate data structures, allowing for efficient verification by reducing data to aggregates for comparison, thereby identifying and addressing potential errors or inconsistencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire data structures are compared for data verification, then data integrity can be ensured, but processing time increases significantly from minutes to hours or days
Solution Approach 1:
The patent extracts aggregate data (sums, counts, averages) from large data structures and compares only these extracted values rather than comparing entire datasets. This extraction of essential verification information reduces processing time from hours/days to minutes while maintaining verification accuracy.
Solution Approach 2:
Instead of performing complete data structure comparison, the patent applies partial action by comparing only aggregate metrics (partial verification). This partial comparison is sufficient to detect data integrity issues without requiring full dataset processing, thereby reducing time consumption significantly.
2Measurement precision
If extensive comparison of entire data structures is performed, then all discrepancies can be detected, but system resources and processing complexity increase
Solution Approach 1:
The patent extracts key aggregate metrics (sums, counts, averages) from complex data structures and compares these extracted values. This approach maintains discrepancy detection capability while avoiding the complexity of comparing entire multi-million row datasets structure-by-structure.
Solution Approach 2:
The patent segments the verification process into two stages: (1) generating and comparing aggregate metrics for quick verification, and (2) optionally drilling down into specific discrepancies only when needed. This segmentation reduces processing complexity by handling the majority of cases at the aggregate level.
3Reliability
If replicated data across multiple databases is verified using traditional methods, then data consistency can be ensured, but the verification process becomes inefficient and error-prone
Solution Approach 1:
The patent extracts aggregate data from replicated datasets across multiple databases and compares these extracted aggregates to verify consistency. This extraction approach maintains reliable consistency verification while dramatically improving efficiency compared to traditional row-by-row comparison methods.
Solution Approach 2:
The patent changes the verification parameter from comparing individual data rows to comparing aggregate parameters (sums, counts, averages). This parameter transformation maintains the ability to detect inconsistencies while improving verification efficiency and reducing errors associated with large-scale data processing.
Data Source
AI summary
Systems and methods are provided for evaluating data included in disparate data structures. One exemplary system comprises a first database including a first data structure having first data, and a second database including a second data structure having second data, where at least some of the second data is duplicative of at least some of the first data. The system also includes a computing device coupled to the first and second databases. The computing device is configured to request a first aggregate of the first data, receive the first aggregate, and load the first aggregate in an aggregate data structure. The computing device is further configured to request a second aggregate of the second data, receive the second aggregate, and load the second aggregate in the aggregate data structure, whereby a comparison of the first and second aggregates may be made.