Factored Word-Sequence Kernel for Text Similarity

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for computing similarity between text sequences are limited by requiring exact matches, which neglects acceptable differences between tokens and thus fail to effectively assess the quality of machine translations and transcriptions.

Innovation Solution

A factored word-sequence kernel that considers multiple linguistic factors, such as surface form, lemma, and part of speech, to compute similarity measures between text spans, allowing for non-identical tokens to be considered similar and incorporating linguistic information to enhance similarity computation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If exact match between tokens is required for similarity computation, then measurement precision is improved, but adaptability deteriorates

Engineering Contradiction:

Improvesimilarity measurement precisionVSAvoidtolerance to token differences

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent changes the parameters used for token comparison from exact surface form matching to multi-factor matching including lemma, part of speech, and morphological features. This allows the system to tolerate variations in token surface forms while maintaining precise similarity measurement through structured factor comparison.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent segments each token into multiple linguistic factors (surface form, lemma, part of speech, morphological features). By comparing these segmented factors independently, the system achieves both precision in measurement and adaptability to different token variations, as each factor can be matched or weighted differently.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If multiple linguistic factors are considered for token matching, then adaptability is improved, but device complexity increases

Engineering Contradiction:

Improvetolerance to token differencesVSAvoidcomputation complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments tokens into discrete linguistic factors that can be independently extracted and compared. This segmentation allows the complexity to be managed by processing each factor separately rather than dealing with the entire token structure at once, reducing computational burden while maintaining adaptability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent allows selective application of linguistic factors based on the specific comparison needs. Not all factors need to be computed or weighted equally in every case, enabling the system to adapt the level of complexity to the task requirements rather than always applying the full multi-factor analysis.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS8077984B2Method for computing similarity between text spans using factored word sequence kernels

Publication Date: 2011.12.13 XEROX CORP

AI summary

A computer implemented method and an apparatus for comparing spans of text are disclosed. The method includes computing a similarity measure between a first sequence of symbols representing a first text span and a second sequence of symbols representing a second text span as a function of the occurrences of optionally noncontiguous subsequences of symbols shared by the two sequences of symbols. Each of the symbols comprises at least one consecutive word and is defined according to a set of linguistic factors. Pairs of symbols in the first and second sequences that form a shared subsequence of symbols are each matched according to at least one of the factors.