Database Match Key Generation for Duplicate Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Database systems face performance issues when updating millions of records due to the inefficiency of matching and updating processes, particularly in identifying duplicate records using matching rules.

Innovation Solution

The implementation of a method that generates match keys based on lookup fields and applies matching rules to identify duplicate candidates, allowing for efficient updating of field values by storing indices as alternative field values and using disjunctive normal form matching rules to quickly identify matching records.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional matching rules are used to identify duplicate records in database systems, then duplicate detection capability is maintained, but system performance deteriorates significantly when updating millions of records

Engineering Contradiction:

Improvedatabase update performanceVSAvoidtime for matching and updating records

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system pre-computes and stores match keys for all records before update operations are performed. These match keys are generated based on lookup fields and disjunctive normal form matching rules in advance, so that during actual update operations, the system can quickly identify duplicate candidates without performing complex matching computations, thereby improving update performance and reducing processing time

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The matching process is divided into distinct segments: first generating match keys from lookup fields, then using these keys to quickly identify candidate duplicates, and finally applying the full matching rules only to these candidates. This segmentation allows the majority of records to be processed efficiently without requiring complete matching rule evaluation, thus improving overall system productivity during updates

Inventive Principle:

Principle #1Segmentation

2Measurement precision

If comprehensive matching rules are applied to all records, then duplicate identification accuracy is maintained, but device complexity increases due to the need to process and compare all field values

Engineering Contradiction:

Improveduplicate detection accuracyVSAvoidcomplexity of matching and updating process

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system extracts only the essential matching information into pre-computed match keys stored in the database. Instead of storing and processing all original field values during matching, the system extracts the critical identification features into compact key representations, reducing the complexity of the matching process while maintaining the ability to accurately identify duplicates when needed

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS10235476B2Matching objects using match rules and lookup key

Publication Date: 2019.03.19 SALESFORCE INC

AI summary

Some embodiments of the present invention include a method for identifying match candidates in a database object and may include generating, by a database system, a match key associated with a lookup field of a database object. The method may further include activating, by the database system, a matching rule associated with the match key, and receiving, by the database system, a request to search for duplicate candidates in the database object. The request may include an input entity. The duplicate candidates in the database object may be identified by using the input entity and the matching rule.