Metadata-Driven Data Reconciliation Using Secure Sample Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data reconciliation methods are time-consuming and inefficient, particularly for large datasets, as they require field-by-field checks and copying sensitive datasets, which is often infeasible and does not scale well.
Innovation Solution
Metadata-driven data reconciliation using random sampling and checksum-based verification, leveraging pre-defined telemetry information and statistical feature verification, with zero configuration and no exposure of sensitive information, to ensure high confidence in data comparison.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional field-by-field data reconciliation is performed on large datasets, then data integrity can be verified, but the process becomes extremely time-consuming and does not scale
Solution Approach 1:
The patent segments the large dataset into smaller samples for verification. Instead of reconciling entire datasets field-by-field, the system extracts and compares representative samples (e.g., 100-1000 records) from each dataset, maintaining verification reliability while dramatically reducing reconciliation time and resource consumption.
Solution Approach 2:
The patent applies partial action by performing reconciliation on a subset (sample) of the data rather than the complete dataset. This sampling approach provides sufficient confidence for data integrity verification without the excessive time cost of full dataset comparison, achieving the right balance between verification thoroughness and efficiency.
2Measurement precision
If full datasets are copied to a comparison engine location, then comprehensive comparison can be performed, but business sensitive data security policies are violated and scalability is limited
Solution Approach 1:
The patent extracts only the necessary comparison elements (sample records and their metadata) from the full datasets, rather than copying entire datasets. This extraction approach maintains comparison accuracy for the sampled portions while eliminating the security risk associated with copying sensitive business data to external locations.
Solution Approach 2:
The system uses metadata and sample-based features as intermediaries to enable comparison without direct access to sensitive full datasets. By working with these intermediary representations, the system achieves comparison functionality while maintaining data security boundaries and avoiding the need to copy protected data.
3Productivity
If metadata-driven sampling and feature verification is used, then reconciliation speed and scalability improve, but complete data coverage is reduced
Solution Approach 1:
The patent changes the parameter of verification from 100% data coverage to statistically significant sample coverage. By adjusting the sample size parameters (e.g., verifying 0.1%-1% of records through feature comparison), the system achieves high productivity while maintaining sufficient data coverage for reliable reconciliation through statistical inference and metadata validation.
Data Source
AI summary
Systems and methods for metadata driven data reconciliation are disclosed. A method may include: (1) identifying origin dataset metadata for an origin dataset; (2) identifying new dataset metadata for a new dataset; (3) comparing the origin dataset metadata to the new dataset metadata; (4) identifying a sample size for the origin dataset, wherein the sample size comprises a number of cells; (5) extracting an origin dataset sample of the sample size of random cells from the origin dataset; (6) generating origin dataset features for the origin dataset sample; (7) generating new dataset features for the new dataset; (8) searching for the origin dataset features in the new dataset features; and (9) returning a matching result in response to the origin dataset features being found in the new dataset features.