Metadata Indexing for Data Record Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face inefficiencies in candidate selection and comparison processes for matching data records, particularly due to the high cost and resource intensity of scanning large datasets for bucket hashes and comparison data, which hampers the efficiency of the matching process.
Innovation Solution
A system utilizing a metadata index with metadata records containing bucket hashes and comparison information, where entity membership is determined through comparisons within the metadata database, reducing the need for multiple database accesses and enhancing processing speed by generating metadata records that include bucket hashes and comparison information for efficient data record matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data management systems scan large datasets to identify candidate records for matching, then comprehensive matching can be achieved, but the processing time and computational resources increase significantly
Solution Approach 1:
The system pre-computes and stores metadata records containing bucket hashes and comparison information for all data records in the database. This preliminary action allows the matching process to quickly retrieve and compare pre-prepared metadata instead of scanning and processing entire data records during matching operations, significantly reducing processing time while maintaining matching accuracy
Solution Approach 2:
The invention extracts essential comparison elements (bucket hashes and comparison information) from complete data records and stores them as separate metadata records. This extraction allows the system to work with compact metadata representations during matching operations, reducing the amount of data that needs to be scanned and processed while preserving the ability to perform accurate comparisons
2Measurement precision
If the system performs multiple database accesses to retrieve bucket hashes and comparison data during matching, then accurate candidate selection can be made, but the number of database I/O operations increases
Solution Approach 1:
The system merges bucket hashes and comparison information into unified metadata records that are stored together in the database. This merging allows the system to retrieve all necessary matching information in a single database access operation rather than performing multiple separate queries to gather bucket hashes, comparison data, and other matching elements, thereby improving processing efficiency while maintaining candidate selection accuracy
3Reliability
If the system stores complete data records for comparison, then comprehensive matching information is available, but the storage space and data transmission requirements increase
Solution Approach 1:
The system extracts only the essential comparison elements (bucket hashes and comparison information) from complete data records and stores them as metadata records. This extraction reduces the volume of stored data while preserving the completeness of matching information needed to accurately determine entity membership and link duplicate records
Data Source
AI summary
A method, apparatus, computer system, and computer program product for managing information. A set of bucket hashes and comparison information for a data record are identified by a computer system. The set of bucket hashes is generated from the comparison information, wherein the set of bucket hashes and the comparison information form a metadata record. A number of candidate metadata records in a metadata database is identified by the computer system using the set of bucket hashes, wherein the number of candidate metadata records comprises a set of candidate bucket hashes and candidate comparison information. An entity membership is identified by the computer system for the data record from a comparison of the comparison information in the metadata record with the candidate comparison information in the number of candidate metadata records.