In-Memory Data Reconciliation via RAM Table Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reconciling large data migrations across different platforms is a time-intensive process, requiring efficient methods to verify data accuracy and readiness for production use.
Innovation Solution
A computer system that compares data across two data sources by storing tables in local RAM, using database connectivity drivers to access and compare data, identifying primary keys, and computing a score based on mismatched cell values to generate a comparison report.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data reconciliation methods are used to compare large data sources, then data accuracy can be verified, but the process becomes extremely time-intensive
Solution Approach 1:
The patent segments the data comparison process by dividing large data sources into smaller manageable tables that can be loaded into RAM. Instead of comparing entire data sources at once, the system breaks down the reconciliation task into multiple smaller table comparisons, enabling parallel processing and reducing overall reconciliation time while maintaining accuracy.
Solution Approach 2:
The patent introduces a new dimension of processing by utilizing RAM memory as an intermediate layer between data sources and the comparison engine. This dimensional change from direct disk-to-disk comparison enables in-memory processing, dramatically accelerating the reconciliation process while preserving data accuracy through systematic sampling and comparison methodologies.
2Productivity
If large amounts of data are loaded into RAM for comparison, then comparison speed increases significantly, but memory resource requirements increase
Solution Approach 1:
The patent applies segmentation by dividing large data sources into smaller tables that fit within available RAM capacity. This allows the system to load multiple smaller tables into memory simultaneously for parallel comparison, achieving high processing speeds without requiring excessive memory resources that would be needed to load entire data sources at once.
Solution Approach 2:
The patent implements partial action by using representative sampling of data tables rather than loading and comparing every single record. The system loads selected tables into RAM for comparison, achieving sufficient reconciliation accuracy through strategic sampling while consuming manageable memory resources, thus balancing speed and resource requirements.
Data Source
AI summary
Computer system compares data across two data sources. Database connectivity drivers access the first and second data sources. RAM receives and stores data from each of the first and second data sources accessed by the drivers. The data from the first data source are stored in a first table in the RAM and data from the second data source are stored in a second table in the RAM. A processor compares data in the first and second tables. The processor is programmed to compare the data by identifying a set of primary keys across the first and second tables. The set of primary keys are a set of one or more columns in the both first and second tables that uniquely identify rows across both the first and second tables. The processor then compares the data in the first and second tables, which comparison can include: determining, based on the identified set of primary keys, a total number of mismatched cell values across the first and second tables; and computing a score for the comparison of the first and second tables, where the score is computed based on the total number of mismatched cell values. The computer system can then generate an electronic comparison report based on the comparing of the first table and second table.