Document Similarity Scoring via Token Pair Matrix Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Electronic marketplaces face challenges in automatically identifying duplicate product descriptions from various vendors, as existing methods rely on manual comparison and are inefficient when dealing with large volumes of data.
Innovation Solution
A method and system for determining the likelihood of document similarity by generating token pairs, calculating similarity scores, and selecting pairs above a threshold to identify matched tokens, providing a similarity score for the documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual comparison methods are used to identify duplicate product descriptions, then precision in identifying duplicates can be maintained, but productivity decreases significantly when dealing with large volumes of data
Solution Approach 1:
The patent replaces manual mechanical comparison with an automated computer-based system that uses tokenization, matrix generation, and similarity scoring algorithms to identify duplicate product descriptions, thereby maintaining precision while dramatically improving productivity
Solution Approach 2:
The patent transforms the comparison task by changing parameters from exact string matching to similarity scoring based on token pair analysis, allowing the system to identify duplicates with high precision while processing large volumes of data automatically
2Productivity
If automated methods are used to identify duplicate product descriptions, then productivity increases, but measurement precision decreases due to difficulty in programmatically analyzing documents
Solution Approach 1:
The patent segments product descriptions into individual tokens and creates a matrix of token pairs, allowing the automated system to analyze each pair's similarity independently and aggregate results, thereby maintaining precision while enabling automated high-volume processing
Solution Approach 2:
The patent introduces similarity scores as an intermediary metric between exact matching and subjective human judgment, enabling automated systems to achieve precision comparable to manual analysis by using calculated similarity thresholds
3Manufacturing precision
If exact matching is used for product descriptions, then manufacturing precision is high, but adaptability decreases when dealing with abbreviations, misspellings, and variations
Solution Approach 1:
The patent changes the matching parameter from exact string equality to similarity scoring based on token pair analysis, enabling the system to adapt to abbreviations, misspellings, and variations while maintaining high precision through configurable similarity thresholds
Data Source
AI summary
A system and method for determining the likelihood of two documents describing substantially similar subject matter is presented. A set of tokens for each of two documents is obtained, each set representing strings of characters found in the corresponding document. A matrix of token pairs is determined, each token pair comprising a token from each set of tokens. For each token pair in the matrix, a similarity score is determined. Those token pairs in the matrix with a similarity score above a threshold score are selected and added to a set of matched tokens. A similarity score for the two documents is determined according to the scores of the token pairs added to the set of matched tokens. The determined similarity score is provided as the likelihood that the first and second documents describing substantially similar subject matter.