De-duplication Module for Master Data Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data management systems face inefficiencies in identifying and removing duplicate master data records across multiple applications, leading to time-consuming and memory-intensive processes.
Innovation Solution
A de-duplication module that separates master data into key and data portions, identifies duplicates, unifies them with a golden record, and removes obsolete records, optimizing the data management system by reducing redundant storage and updating processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data management systems execute regular operations to consolidate data and remove duplicate records, then data accuracy is improved, but processing time and memory consumption increase significantly
Solution Approach 1:
The system performs de-duplication operations proactively during data ingestion or at scheduled intervals rather than waiting for regular consolidation cycles. The de-duplication module identifies and merges duplicate records before they accumulate, preventing data quality degradation while avoiding the need for time-consuming periodic full-system scans.
Solution Approach 2:
The patent extracts the de-duplication function as a separate, dedicated module that operates independently from the main data consolidation process. This extracted module focuses specifically on identifying and merging duplicate records using efficient algorithms, separating this time-consuming task from other data processing operations to improve overall system efficiency.
2Reliability
If conventional data management systems execute regular operations to consolidate data and remove duplicate records, then data accuracy is improved, but memory consumption increases significantly
Solution Approach 1:
The de-duplication module segments the data processing task by dividing the dataset into manageable chunks or partitions. It processes records in batches rather than loading entire datasets into memory simultaneously, using indexing and hashing techniques to identify duplicates across partitions with minimal memory footprint.
Solution Approach 2:
The system introduces intermediate data structures such as hash indexes or bloom filters that act as mediators between the raw data and the de-duplication logic. These intermediaries enable efficient duplicate detection with constant or logarithmic memory access, avoiding the need to store and compare all record pairs in memory.
3Adaptability or versatility
If multiple applications create independent data records for the same entity, then application autonomy is maintained, but data redundancy increases
Solution Approach 1:
The de-duplication module serves as a universal service that multiple applications can utilize simultaneously. Each application maintains its autonomy to create records independently, but the shared de-duplication module ensures that duplicates across all applications are identified and merged, providing a multi-functional solution that preserves application independence while eliminating redundancy.
Solution Approach 2:
The system implements feedback mechanisms where the de-duplication module continuously monitors data across applications and provides feedback about duplicate detections. When duplicates are identified, the system feeds back merge recommendations to the applications, allowing them to maintain autonomy while progressively reducing redundancy based on system-generated insights.
Data Source
AI summary
A method and system including at least one application, wherein a data storage is associated with each application; a de-duplication module; a de-duplication processor in communication with the de-duplication module and operative to execute processor-executable process steps to cause the system to: receive a data storage for each of the at least one application, wherein a first data storage and a second data storage include a same record or the one data storage includes two or more of the same record; separate the records into a key portion and a data portion; assign an internal key to each record, wherein the internal key is stored in both the key portion and the data portion for that record; calculate a golden record including golden record values; assign a golden record internal key to the golden record; replace each internal key in the key portion with the golden record internal key; and remove the data portion of each data record where the internal key in the data portion is different from the internal key in the key portion. Numerous other aspects are provided.