Archive Management for Approximate String Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in reconciling datasets with fields having equivalent values that are not exactly the same, requiring efficient methods for approximate string matching to perform operations like join and rollup, especially when dealing with varied spellings, misspellings, and inconsistent formatting across different data sources.
Innovation Solution
A method for managing an archive that processes records to generate string representations, creates close representations by deleting characters, and stores entries representing potential approximate matches, using significance values and scores to quantify the quality of matches, allowing for efficient comparison and grouping of records with tolerant string matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If exact string matching is used for database operations, then matching precision is high, but the system cannot handle varied spellings and inconsistent formatting across different data sources
Solution Approach 1:
The patent applies preliminary action by pre-computing close representations (including deletions, substitutions, and transpositions of characters) for all strings in the database before performing approximate matching operations. This preprocessing step creates an archive of variant forms that enables efficient fuzzy matching without sacrificing accuracy, as the system can quickly compare pre-generated close representations against query strings.
Solution Approach 2:
The patent introduces close representations as an intermediary between exact string matching and fuzzy matching. By generating and storing intermediate forms (strings with one or more character modifications) in an archive, the system mediates between precise matching requirements and the need to handle spelling variations, allowing accurate approximation through comparison of these intermediate forms.
2Adaptability or versatility
If approximate string matching is implemented to handle inconsistent data, then adaptability improves, but computational complexity and processing time increase
Solution Approach 1:
The patent reduces computational complexity during query execution by performing the computationally intensive generation of close representations in advance and storing them in an archive. During actual matching operations, the system simply compares query strings against pre-computed close representations, dramatically reducing the computational burden at query time while maintaining comprehensive approximate matching capability.
Solution Approach 2:
The patent extracts the computationally expensive operations (generating all possible close representations through deletions, substitutions, and transpositions) and separates them from the query execution process. By extracting these operations and performing them beforehand during data loading or indexing, the system eliminates the need to perform complex computations during each matching operation, significantly improving runtime performance.
3Reliability
If all possible close representations are generated for every string, then matching completeness is high, but the quantity of data stored in the archive becomes large
Solution Approach 1:
The patent applies local quality by selectively generating close representations based on the specific characteristics and requirements of each string or data context. Rather than uniformly generating all possible variants for every string, the system can adjust the level of close representation generation based on local factors such as string length, frequency of occurrence, and importance weight, thereby reducing overall archive size while maintaining reliability for critical matches.
Solution Approach 2:
The patent implements partial action by generating close representations with a limited number of character modifications (e.g., one or two deletions, substitutions, or transpositions) rather than exhaustively generating all possible variants. This partial generation approach achieves sufficient matching completeness for practical applications while dramatically reducing the quantity of data stored in the archive compared to exhaustive generation.
4Measurement precision
If significance values and scores are computed for all string pairs, then matching accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The patent computes significance values and match scores in advance during the indexing or data loading phase, storing these pre-computed scores in the archive alongside close representations. During query execution, the system retrieves pre-computed scores rather than calculating them in real-time, significantly reducing processing time while maintaining high matching accuracy through the use of these pre-assessed quality metrics.
Solution Approach 2:
The patent creates and stores copies of match quality assessments (significance values and scores) alongside the close representations in the archive. By copying and storing these quality metrics in advance, the system eliminates the need to re-compute them during each query, reducing processing time while preserving the accuracy information needed for reliable approximate matching.
Data Source
AI summary
In one aspect, in general, a method is described for managing an archive for determining approximate matches associated with strings occurring in records. The method includes: processing records to determine a set of string representations that correspond to strings occurring in the records; generating, for each of at least some of the string representations in the set, a plurality of close representations that are each generated from at least some of the same characters in the string; and storing entries in the archive that each represent a potential approximate match between at least two strings based on their respective close representations.