String Similarity Scoring Using Frequency Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional string comparison algorithms struggle to accurately distinguish between similar and dissimilar name strings, leading to increased false positives due to their inability to account for frequency of occurrence and cultural or regional variations, especially with short character strings.

Innovation Solution

A method that utilizes frequency data to adjust similarity scores by determining the occurrence frequency of character strings within a database, thereby enhancing the accuracy of string comparisons by differentiating between similar and dissimilar strings.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional edit distance algorithms are used for string comparison, then short name strings can be processed, but the accuracy of similarity scoring deteriorates due to inability to distinguish between related and unrelated names

Engineering Contradiction:

Improveprocessing capability for short stringsVSAvoidsimilarity scoring accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent changes the parameter used for string comparison from purely character-based edit distance to a hybrid approach that incorporates frequency data. By introducing frequency as an additional parameter, the system can distinguish between common short names (like 'Bill' and 'Jill') and actual typos (like 'Patricia' and 'Patircia'), thereby improving measurement precision while maintaining processing capability

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent introduces frequency data as an intermediary element that mediates between the raw string comparison and the final similarity score. This intermediary provides contextual information about how common certain name patterns are, allowing the system to adjust similarity scores appropriately based on statistical likelihood rather than just character matching

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If string comparison allows single character differences to find matches, then more potential matches are found, but false positives increase significantly

Engineering Contradiction:

Improvenumber of matches foundVSAvoidfalse positive rate

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent implements feedback by using frequency data to adjust and refine similarity scores. The system compares the frequency of candidate matches against the frequency of the original string, and this feedback mechanism penalizes matches that are statistically unlikely to be related, thereby reducing false positives while maintaining high productivity

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent makes the similarity scoring dynamic by adjusting it based on frequency ratios. Rather than using a fixed threshold, the system dynamically modifies scores based on how common or rare the compared strings are in the database, allowing flexible adaptation to different naming patterns and reducing false positives across diverse name types

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS9747274B2String comparison results for character strings using frequency data

Publication Date: 2017.08.29 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A similarity between character strings is assessed by identifying first and second character strings as candidate similar character strings, determining a frequency of occurrence for at least one of the first and second character strings from a collection of character strings, and designating the first and second character strings as similar based on the determined frequency of occurrence.