Data De-duplication System for Accurate Reporting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data warehouse systems face challenges in managing duplicate dimension data records from multiple sources, leading to inaccurate reports and calculations, as they lack efficient methods to identify and merge duplicates without requiring significant modifications to source applications and databases.
Innovation Solution
A data de-duplication system that generates de-duplication instructions to identify and merge duplicate dimension data records, creating a single best record (SBR) and adjusting foreign keys to reference this record, thereby eliminating duplicates within the data warehouse without modifying source applications or databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is loaded from multiple source databases into a data warehouse, then data integration and reporting capability are improved, but duplicate dimension data records are created leading to inaccurate reports and calculations
Solution Approach 1:
The system performs preliminary de-duplication processing during data loading by applying de-duplication instructions to identify and merge duplicate dimension records before they can cause reporting inaccuracies. This preliminary action prevents the accumulation of duplicates that would otherwise corrupt report accuracy.
Solution Approach 2:
The patent introduces de-duplication instructions as an intermediary mechanism between data loading and reporting. These instructions act as a mediator that identifies duplicate dimension records and directs the system to consolidate them, thereby preserving report accuracy while maintaining multi-source data integration.
2Manufacturing precision
If de-duplication processing is implemented to eliminate duplicate records, then data accuracy is improved, but system complexity increases due to additional processing steps
Solution Approach 1:
The system implements self-service de-duplication by automatically applying de-duplication instructions to identify and merge duplicate dimension records without requiring manual intervention. The system serves itself by autonomously detecting duplicates and consolidating records, thereby improving data accuracy while minimizing the operational complexity burden.
3Reliability
If traditional de-duplication methods are used, then duplicate records can be identified, but significant modifications to source applications and databases are required
Solution Approach 1:
The patent extracts the de-duplication logic from the source applications and databases and implements it separately as de-duplication instructions applied during data loading. By taking out the de-duplication function from the source systems, the invention achieves reliable duplicate identification without requiring modifications to the source applications or their database structures.
Data Source
AI summary
A data de-duplication system is provided that supports the loading and integration of data from multiple data sources. The data de-duplication system identifies and merges duplicate dimension data records that describe the same entity by creating a single dimension data record that is identified as a single best record (“SBR”). The data de-duplication system further adjusts foreign keys that reference the duplicate dimension data records so that the foreign keys correctly reference the merged dimension data record (i.e., the SBR).