Record Attribute Embeddings for Efficient Match Candidate Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing methods, such as hash-based bucketing, require extensive comparisons to match records, leading to inefficiencies and increased storage requirements.
Innovation Solution
Utilizing trained embedding generation artificial intelligence models to generate record attribute embeddings, group them based on similarity, and compare records within these groups to identify matches, thereby optimizing candidate selection and reducing storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hash-based bucketing technique is used for candidate selection, then the total number of comparisons is reduced, but the storage requirements and device complexity increase
Solution Approach 1:
The patent extracts only the essential features needed for candidate selection (name, address, date of birth, identifier) and uses hashing on these extracted features to create compact bucket identifiers. This selective extraction reduces the storage burden while maintaining the ability to group similar records efficiently.
Solution Approach 2:
The patent transforms record attributes into hash values, changing the parameter representation from original attribute values to compressed hash codes. This parameter transformation enables efficient grouping and comparison while significantly reducing the storage space required for candidate selection structures.
2Measurement precision
If extensive comparison of different record attributes is performed to match records, then matching accuracy is improved, but the processing time increases
Solution Approach 1:
The patent segments the record matching process into two distinct phases: candidate selection (grouping by hash values of key attributes) and detailed comparison (verifying matches within candidate pairs). This segmentation allows extensive attribute comparison to be performed only on a small subset of candidate records rather than all record pairs, thereby maintaining accuracy while reducing processing time.
Solution Approach 2:
The patent performs preliminary grouping of records into buckets based on hash values of key attributes before conducting detailed attribute comparisons. This preliminary action filters out non-matching records in advance, so that extensive comparisons are only performed on records that are likely matches, significantly reducing the overall processing time while preserving matching accuracy.
3Productivity
If hash-based candidate selection is used to group records by exact hash value equality, then comparison efficiency is improved, but the quality of candidate selection deteriorates
Solution Approach 1:
The patent uses hash functions with universal properties that distribute similar inputs into the same buckets while also handling variations in data representation. The hash-based grouping serves multiple functions: it groups potential matches efficiently, handles different data formats (case variations, formatting differences), and maintains comparison efficiency, thereby achieving both productivity and candidate selection quality.
Data Source
AI summary
One or more trained embedding generation artificial intelligence models are executed to generate a plurality of record attribute embeddings. The plurality of record attribute embeddings represents a plurality of attributes of data of a plurality of records. Grouping of the plurality of record attribute embeddings is performed. The grouping of a record attribute embedding includes grouping attribute values of the record attribute embedding into one or more groups of attribute values. The performing grouping provides a plurality of groups of attribute values for the plurality of record attribute embeddings. Selected records are compared to provide a set of matched records. The comparing, based on a group of attribute values, includes comparing records that include one or more attribute values grouped in the group of attribute values providing a subset of matched records of the set of matched records. The set of matched records is stored in an accessible computer location.