Pre-import Duplicate Record Matching in Data Hubs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for identifying duplicate records in data hubs are inefficient, as they require manual identification and merging after records are imported, leading to increased downtime and inconsistent data views, especially when dealing with large batches of records.
Innovation Solution
Implementing a technique that selects a match rule for a batch of records to be imported into a data hub, using match criteria and operators to identify duplicates before importation, allowing for pre-import matching and merging or discarding of duplicates, with user interface support for review and operation on identified records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual identification and merging of duplicate records is performed after importation, then data consistency is maintained, but processing time and downtime increase significantly
Solution Approach 1:
The patent applies preliminary action by performing duplicate record identification and merging operations before data is imported into the production data hub. The system pre-processes incoming data batches, identifies potential duplicates using match rules, and merges them in advance, so that only cleaned data is loaded into the production environment. This eliminates post-import processing time and downtime while maintaining data consistency.
2Reliability
If production environment is brought down for duplicate removal, then data consistency is ensured, but data hub availability decreases
Solution Approach 1:
The system performs duplicate identification and merging in a pre-import stage before data enters the production environment. This allows the production data hub to remain continuously available and operational throughout the duplicate removal process, as all cleansing operations are completed in advance on separate data batches.
Solution Approach 2:
The patent introduces an intermediary pre-processing stage or staging area where duplicate removal operations are performed on incoming data batches before they are loaded into the production data hub. This intermediary layer acts as a buffer, allowing data cleansing to occur without impacting the availability of the production environment.
3Ease of operation
If ex post duplicate identification is performed, then manual merging can be carried out, but the process becomes inefficient and time-consuming
Solution Approach 1:
The system automatically performs duplicate identification and merging operations in advance before data importation. Match rules are applied to incoming data batches to identify potential duplicates, and the system automatically merges them using defined criteria. This preliminary automated processing eliminates the need for manual identification and merging operations, significantly improving processing efficiency while maintaining operational simplicity.
Data Source
AI summary
Techniques for identifying duplicate records in data to be imported into a data hub. In one set of embodiments, a batch of records to be imported into a data hub is received and a match rule for the batch of records is selected. The match rule includes a set of match criteria for matching records in the inbound batch to records stored in the data hub. A matching process is then carried out, prior to importing the batch of records, to determine whether any of the records in the batch are duplicates of records stored in the data hub. By identifying potential duplicate records before (rather than after) importation, the problems associated with prior art duplicate identification techniques are minimized or avoided. In a particular embodiment, an import workbench application comprising a plurality of user interface screens is provided for aiding a user in managing the matching process.