Similar Data Record Matching With Q-Gram Filtering

The patent segments the fuzzy matching process into distinct phases: candidate generation using q-gram indexing, filtering using multiple criteria (length filter, q-gram filter, shared character count filter), and final verification using edit distance calculation. This segmentation allows the system to quickly eliminate non-matching records before performing computationally expensive operations, thereby reducing overall processing time while maintaining accuracy.

The patent applies preliminary filtering actions before the main matching computation. Specifically, it pre-processes records by generating q-grams, calculating length metrics, and creating candidate sets that are likely to match. These preliminary actions reduce the search space significantly, allowing the system to handle large datasets efficiently without sacrificing matching accuracy.

The patent extracts only the necessary components for matching from the full dataset. Instead of loading and comparing all possible record pairs, it extracts candidate records based on q-gram similarity and other filtering criteria. This extraction approach reduces memory requirements by focusing computational resources only on potentially matching records rather than the entire dataset.

The patent segments the data processing into manageable chunks through candidate generation and filtering stages. By dividing the large dataset into smaller candidate sets that pass through multiple filtering layers, the system reduces the memory footprint at any given time, allowing accurate fuzzy matching on large datasets without requiring proportional memory increases.

The patent implements a dynamic, multi-stage filtering process that adapts the matching criteria at different stages. The system starts with lenient q-gram-based candidate generation to capture spelling variations, then progressively applies stricter filters (length filter, q-gram filter, shared character count filter) to eliminate false positives. This dynamic approach maintains adaptability to spelling variations while progressively reducing false positive rates.

The patent incorporates feedback mechanisms through its multi-stage filtering process. Each filtering stage provides feedback to the next stage by eliminating records that fail to meet specific criteria. The shared character count filter and edit distance verification use feedback from previous stages to further refine the candidate set, thereby reducing false positives while maintaining the ability to handle spelling variations.