Text Similarity Detection Using Word-Level Levenshtein Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for merging different versions of electronic documents, especially in software engineering, often require manual intervention due to moved or modified text portions, which are not easily identifiable through automatic comparison.
Innovation Solution
A method that compares text portions on a word basis using a modified Levenshtein algorithm to determine if a first text portion is included in a second, considering amendments, by calculating the minimum number of word insertions, deletions, and substitutions, and applying threshold values to assess similarity and inclusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a character-based comparison algorithm is used to identify moved text portions, then measurement precision is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments text into words rather than comparing character-by-character. The Levenshtein algorithm is applied at the word level, dividing the text comparison task into smaller discrete units (words) that can be processed more efficiently while still capturing semantic similarity.
Solution Approach 2:
The patent changes the granularity parameter of text comparison from character-level to word-level. This parameter change maintains the ability to detect similarity (since words are the meaningful units of text) while dramatically reducing the computational complexity from O(n*m) character comparisons to fewer word-level comparisons.
2Measurement precision
If text portions are compared on a character basis to detect moved text, then detection accuracy is improved, but device complexity increases
Solution Approach 1:
The algorithm segments text into words using delimiters (spaces, punctuation) and compares these word units rather than individual characters. This segmentation reduces the number of comparison elements from potentially thousands of characters to fewer hundred or thousand words, simplifying the computational task while preserving detection accuracy.
Solution Approach 2:
The patent replaces the mechanical character-by-character comparison process with a more efficient word-level comparison system. By substituting the fine-grained character comparison mechanism with coarser word-level comparison, the system achieves similar detection goals with reduced computational overhead and simpler processing logic.
3Measurement precision
If manual intervention is used to identify moved text portions, then measurement precision is improved, but productivity deteriorates
Solution Approach 1:
The system performs self-service by automatically detecting moved and modified text portions through the word-based Levenshtein algorithm. Instead of requiring users to manually identify changes, the algorithm autonomously compares text portions, calculates similarity metrics, and identifies inclusions, enabling automatic merge operations without human intervention while maintaining high accuracy.
Solution Approach 2:
The patent introduces configurable threshold parameters that allow the system to automatically determine when text portions are similar enough to be considered included. By changing from manual judgment to automated threshold-based decision making, the system achieves both high productivity (automatic operation) and maintained precision (configurable accuracy standards).
Data Source
AI summary
The invention refers to a method executed by a computer for automatically determining whether or not a first text portion (1) is to be considered as included in a second text portion (2), wherein it may be included in an unamended or amended form, both text portions being electronically coded and being structured in a respective number of one or more lines (4; 5), the method characterised by the steps of selecting a line Li in the first text portion and a line Zj in the second text portion, the line Li including at least one, two, three or more word/s (3) and the line Zj including at least one, two, three or more word/s; applying a predetermined algorithm to the tuple consisting of the lines Li and Zj, the algorithm comparing each word in the line Li with one or some or each word/s in the line Zj to determine whether or not the compared words are equal, and calculating, by utilising whether or not the compared words are equal, at least one result value; and determining, by utilising the at least one result value, whether or not the first text portion is to be considered as included in the second text portion.