Dynamic Field Weighting for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data-deduplication methods rely on static weights for fields, leading to misidentification of duplicate and unique records, increasing the number of records that require manual classification by data stewards and resulting in inefficiencies and higher costs.
Innovation Solution
Dynamic weighting of field values based on their content, using a composite score calculated from weighted matches to determine record duplication, with the aid of algorithms like Support Vector Machines to refine weights and improve classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If static weights are used for fields in probabilistic matching, then the system is simple to implement, but the accuracy of duplicate record identification deteriorates
Solution Approach 1:
The patent applies dynamics by transitioning from static field weights to dynamic weights that are calculated based on the actual values within records. The weight for each field is determined by comparing the distribution of values across the dataset, allowing the system to adapt to the specific characteristics of the data being processed. This resolves the contradiction by maintaining computational simplicity while significantly improving identification accuracy through value-based weight adjustment.
Solution Approach 2:
The patent changes the parameter of field weights from fixed static values to dynamic values that vary based on the data content. By calculating weights as a function of value distribution statistics (such as frequency, entropy, or uniqueness metrics), the system adapts its matching sensitivity to the actual data characteristics. This parameter transformation enables accurate duplicate identification without requiring complex manual configuration.
2Productivity
If static weights are used for fields, then the processing speed is fast, but the number of misidentified records increases
Solution Approach 1:
The system maintains fast processing by using dynamic weights that are calculated once from value distribution statistics and then applied consistently during matching operations. The dynamic nature of the weights (based on data characteristics) improves reliability, while the efficient calculation and reuse of these weights preserve processing speed. This resolves the contradiction between speed and accuracy.
Solution Approach 2:
The patent performs preliminary calculation of field weights based on value distribution analysis before the actual duplicate identification process. By pre-computing the optimal weights from the dataset characteristics and storing them for reuse, the system eliminates the need for repeated complex calculations during matching, thus maintaining high processing speed while achieving accurate classification through data-adaptive weights.
3Use of energy by moving object
If static weights are used for fields, then the system requires minimal computational resources, but the number of records requiring manual classification increases
Solution Approach 1:
The patent changes the computational approach from using fixed weights to calculating weights based on value distribution parameters. The weight for each field is derived from statistical measures of the data (such as frequency distributions, entropy, or uniqueness ratios), which requires additional computational resources but significantly improves classification accuracy by adapting to the specific data characteristics rather than using generic fixed weights.
Data Source
AI summary
Data-deduplicating includes comparing a first record of a data-store with a second record of the data-store but instead of using a static weight for a field, the present data-deduplicating dynamically assigns a first weight for the first score to generate a first weighted score, wherein the first weight is based on one or both of the first value or the second value; and assigns a second weight for the second score to generate a second weighted score. A composite score is calculated based on the first weighted score and the second weighted score; and it is determined whether or not the first record and the second record are duplicate records, based on the composite score.