Multi-dimensional Record Indexing via Locality Sensitive Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for identifying similar records in large datasets are computationally expensive and may not be complete, as they often rely on full pair-wise similarity comparisons or single-dimensional analyses, which can lead to reduced accuracy and efficiency in identifying multi-dimensional similar records.
Innovation Solution
The method constructs blocking keys for each record by combining attributes, determines a locality sensitive hash based on these keys, and indexes records using this hash to efficiently identify similar records across multiple dimensions, ensuring completeness and reasonable accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full pair wise similarity comparison is made to identify similar records, then completeness of results is improved, but computational cost increases prohibitively
Solution Approach 1:
The patent segments the similarity comparison process into multiple dimensions, where each dimension is analyzed separately to identify similar records. This divides the computationally expensive full pairwise comparison into smaller, more manageable dimensional analyses, reducing overall computational cost while maintaining result completeness.
Solution Approach 2:
The patent transitions from single-dimensional analysis to multi-dimensional analysis by examining records across multiple dimensions separately. This dimensional approach allows the system to identify similar records more efficiently by leveraging the structure of multi-dimensional data, thereby reducing computational requirements compared to exhaustive pairwise comparison.
2Productivity
If records are indexed according to a key to reduce computational complexity, then efficiency is improved, but information loss occurs reducing accuracy
Solution Approach 1:
The patent segments the record attributes into multiple dimensions, creating indexes for each dimension separately. This segmentation allows efficient querying along individual dimensions while preserving the完整性 of the original multi-dimensional information, thereby maintaining accuracy without sacrificing efficiency.
Solution Approach 2:
The patent changes the parameter representation by creating dimension-specific indexes that transform the original multi-dimensional records into dimensionally-decomposed index structures. This parameter transformation enables efficient similarity search while preserving the essential characteristics needed for accurate matching.
3Productivity
If single dimensional analysis is used to identify similar records, then computational efficiency is improved, but accuracy decreases due to ignoring other dimensions
Solution Approach 1:
The patent implements a universal multi-dimensional indexing framework where each dimension has its own index, but all dimensions work together to identify similar records. This multi-functional approach allows the system to leverage efficiency from individual dimensional analysis while incorporating information from all dimensions to maintain high accuracy in similarity identification.
Solution Approach 2:
The patent explicitly transitions from single-dimensional to multi-dimensional analysis by creating and utilizing indexes across multiple dimensions. This dimensional expansion allows the system to maintain computational efficiency through indexed access while improving accuracy by considering similarity across all relevant dimensions simultaneously.
Data Source
AI summary
A method, apparatus and computer program product are provided to index a plurality of records based upon the similarity of the records along a plurality of dimensions. In the context of a method, one or more blocking keys are constructed for each of the plurality of records. A respective blocking key represents a string representation that combines components derived from attributes of a respective record, such as by concatenating components derived from attributes of the respective record. For a respective blocking key, the method determines a locality sensitive hash based upon the string representation of the respective blocking key. The method also includes indexing the respective record based upon the locality sensitive hash.