Big Data Reconciliation via Metadata Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data reconciliation algorithms are sub-optimal for big data environments, struggling to efficiently identify and locate discrepancies between large data sets within a short time frame, especially in systems handling data sizes exceeding several terabytes.
Innovation Solution
The method involves receiving and reconciling large data sets by identifying metadata, constructing SQL queries to join and compare the data sets, and dynamically executing these queries to identify and display discrepancies, allowing for efficient data reconciliation in big data environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data reconciliation algorithms are used to compare large data sets, then data transfer verification can be performed, but the process takes too long and cannot complete within a predetermined time period
Solution Approach 1:
The patent divides the large data sets into multiple blocks or segments that can be processed independently and in parallel. Instead of comparing entire terabyte-sized data sets sequentially, the system segments the data into manageable chunks that can be reconciled simultaneously across multiple processing units, dramatically reducing the overall reconciliation time while maintaining verification reliability
Solution Approach 2:
The system performs preliminary actions by generating and comparing metadata (such as hashes, checksums, or structural descriptors) before conducting full data comparison. This preliminary metadata comparison quickly identifies potential discrepancies without requiring complete data set comparison, enabling the system to determine within the predetermined time period whether differences exist and then focus detailed analysis only on identified discrepancy locations
2Measurement precision
If conventional data reconciliation methods are applied to terabyte-sized data sets, then discrepancies can be identified, but the system cannot locate the specific positions of differences efficiently
Solution Approach 1:
The patent introduces metadata as an intermediary layer between the raw data and the comparison process. This metadata acts as a mediator that encodes positional and structural information about the data sets, allowing the system to quickly identify not just that differences exist but precisely where they are located. The metadata serves as an index or map that guides the discrepancy localization process without requiring exhaustive search through the entire terabyte-sized data
3Reliability
If conventional algorithms process big data environments, then complete data comparison is achieved, but the complexity and resource requirements become unsustainable
Solution Approach 1:
The system creates and processes copies of metadata representations rather than manipulating the actual terabyte-sized data sets during comparison. By working with compact metadata copies that capture essential structural and verification information, the system maintains reconciliation accuracy while dramatically reducing the computational complexity and resource requirements. The metadata copies serve as lightweight proxies that enable complex comparison operations without requiring proportional resources
Data Source
AI summary
Apparatus and methods for reconciling data in a big data environment is provided. Methods may receive a first data set and a second data set for reconciliation. Methods may identify a first set of metadata associated with the first data set. Methods may identify a second set of metadata associated with the second data set. Methods may include a data reconciliation algorithm. The algorithm may compare the first set of metadata to the second set of metadata to obtain a subset of data found within the first data set and a subset of data found within the second data set that are joinable. Methods may dynamically construct one or more SQL queries to identify any discrepancies between the first data set and the second data set. Methods may execute the one or more SQL queries that identify any discrepancies between the first data set and the second data set.