Distributed Data Validation via Checksums and Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data replication validation techniques are inefficient for large data sets in heterogeneous database systems, especially when metadata is limited or not available, leading to difficulties in row-by-row matching and high false-positive rates.
Innovation Solution
A system and method for validating a target data store by loading both source and target data tables into a distributed memory, using checksums, aggregation summaries, and rules-based row-by-row comparisons to ensure data accuracy, with engines for checksum calculation, data aggregation, rule generation, and validation, executed in parallel across multiple computing systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If row-by-row matching and comparing of records is performed across source and target databases, then data validation accuracy is improved, but processing efficiency deteriorates due to the large size of distributed data sets
Solution Approach 1:
The patent divides the large distributed data sets into smaller partitions that can be processed in parallel across multiple computing systems. Each partition is validated independently, allowing row-by-row matching to be performed on manageable subsets rather than the entire data set at once, thus maintaining accuracy while improving processing efficiency through parallelization.
2Reliability
If automated row-by-row matching is performed in distributed environments, then data validation is achieved, but processing difficulty increases when records are not stored or indexed in any particular order
Solution Approach 1:
The patent performs preliminary indexing and sorting operations on the distributed data sets before validation begins. Records are pre-processed to establish a consistent ordering mechanism, which simplifies the subsequent row-by-row matching process. This preliminary action reduces the complexity of handling unsorted distributed records while ensuring reliable validation.
3Quantity of substance
If data sets are partitioned across database systems, then storage capacity is improved, but row-by-row matching efficiency deteriorates
Solution Approach 1:
The patent merges the partitioned data sets from multiple database systems into a unified processing framework. By coordinating the validation process across distributed partitions and using centralized checksum validation, the system maintains the storage benefits of partitioning while achieving efficient matching through coordinated processing and reduction operations.
4Adaptability or versatility
If record identifiers are independently generated by individual database systems, then system autonomy is improved, but row-by-row matching accuracy deteriorates due to false-positives
Solution Approach 1:
The patent introduces checksums as an intermediary validation mechanism. Instead of relying on independently generated record identifiers for matching, the system computes checksums over data partitions and uses these checksums as the primary validation criterion. This intermediary approach maintains system autonomy for identifier generation while eliminating false-positives in validation by using a deterministic checksum-based comparison method.
Data Source
AI summary
Embodiments of the present disclosure include systems and methods for validating a target data table based on a source data table. A distributed memory comprises a plurality of computing systems, each storing at least a portion of the source data table and the target data table in local memory. Processing engines can be efficiently executed on each of the plurality of computing systems to perform comparison functions based on in-memory data. A checksum comparison engine is configured to compare source and target checksums. A data aggregation engine is configured to produce column-based aggregation summaries. A rule generation engine is configured to generate validation rules for checking by a validation engine.