Parallel Data Matching Engine for Real-Time De-duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in creating and maintaining a unified data repository due to duplicate data records, which wastes storage resources and affects accuracy and performance, lacking effective mechanisms for real-time data matching and de-duplication.
Innovation Solution
A system and method that employs a parallel processing engine to match records based on predefined conditions, performing actions to eliminate duplicates and maintain a unique data repository in real-time, using a load repository, master repository, and detail repository with unique identifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data matching and de-duplication processes are performed using traditional sequential processing methods, then system complexity is reduced, but processing speed and real-time capability deteriorate
Solution Approach 1:
The patent segments the data matching process into multiple parallel processing threads, where each thread independently processes different portions of the data stream. This allows simultaneous execution of matching operations against master data, image data, and self-joins without requiring a single complex sequential processor, thereby increasing speed while managing complexity through modular organization
Solution Approach 2:
The patent merges multiple processing functions into a unified parallel processing framework that handles master data matching, image data matching, and self-join operations concurrently. By combining these previously separate sequential processes into parallel execution units, the system achieves real-time processing capability without proportionally increasing overall system complexity
2Measurement precision
If comprehensive matching processes are performed on all data records, then data accuracy and de-duplication quality improve, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary filtering and pre-processing of data records before the main matching operations. By preparing and organizing data in advance (segmenting into load repository, master repository, and image repository), the system reduces the computational burden during actual matching, allowing comprehensive accuracy checks to be performed faster
Solution Approach 2:
The patent implements a multi-pass matching approach where not all records require the full sequence of matching operations. Records are processed through selective paths based on initial filtering, applying comprehensive matching only where necessary. This partial action approach maintains high accuracy for critical matches while reducing overall processing time for the entire data set
3Speed
If multiple repositories store separate data copies, then data accessibility and retrieval speed improve, but storage space consumption and data redundancy increase
Solution Approach 1:
The patent introduces an image repository as an intermediary layer between the master repository and the load repository. This intermediary stores processed matching results and intermediate data, allowing fast retrieval without duplicating entire master data sets across multiple repositories. The image repository acts as a buffer that speeds up access while minimizing redundant storage through selective caching of only necessary data
4Reliability
If real-time data matching is implemented, then duplicate elimination and data quality improve, but system resource consumption and operational costs increase
Solution Approach 1:
The patent implements periodic batch processing of data streams rather than continuous processing of every single record in real-time. Data is collected and processed in periodic batches through the parallel processing engine, maintaining high data quality through comprehensive matching while reducing peak resource consumption by spreading computational load over time intervals
Solution Approach 2:
The patent dynamically adjusts matching parameters and processing intensity based on data characteristics and system load. By changing parameters such as matching thresholds, processing depth, and parallel thread activation levels, the system maintains high data quality when needed while conserving resources during lower-priority operations, optimizing the balance between reliability and resource usage
Data Source
Figure 1
Figure 2
Figure 2A
AI summary
A system and method for creating and maintaining a unique data repository (102) comprising a matching process based on a set of predefined matching conditions and thereon performing an action type corresponding to the outcome of matching process. The present disclosure provides for real time data de-duplication and updating of the unique data repository to obtain a unified view of unique and matching records.