Document Similarity Identification Using Compound Likelihood and Descriptiveness Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for identifying similar documents, such as those using term frequency-inverse document frequency (Tf-idf), face challenges in accurately weighting words and handling misspellings or meaningful pairs, leading to suboptimal results.
Innovation Solution
A system and method that utilize a compound likelihood database and a descriptive database to calculate prominence and descriptiveness scores for words and pairs of consecutive words, enabling a more accurate identification of similar documents by considering the relative weights and probabilities of words and their combinations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If term frequency-inverse document frequency (Tf-idf) is used to weight word occurrences, then the identification process is simple and fast, but rare words are incorrectly assumed to be informative and common words are incorrectly assumed to be uninformative
Solution Approach 1:
The patent changes the parameters used for word weighting from simple frequency counts to a multi-factor scoring system that includes descriptiveness scores, compound likelihood probabilities, and prominence scores. This transforms the identification process from relying solely on raw frequency data to using normalized, context-aware metrics that better reflect actual word importance.
Solution Approach 2:
The patent introduces intermediary databases (compound likelihood database and descriptiveness database) that act as mediators between the raw text and the similarity identification process. These databases pre-process and store contextual information about word relationships, allowing the system to make more accurate assessments without sacrificing identification speed.
2Device complexity
If traditional text comparison methods are used, then the system structure is simple, but the ability to handle misspellings and meaningful word pairs is insufficient
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing descriptiveness scores and compound likelihood probabilities in databases before the actual similarity identification process. This allows the system to handle misspellings and word pairs more reliably during the identification phase without adding complex real-time processing logic.
Solution Approach 2:
The patent combines multiple types of information (descriptiveness scores, compound likelihood probabilities, prominence scores) into a composite scoring system. This composite approach allows the system to handle various text variations and complexities more effectively than any single metric could alone.
3Measurement precision
If compound likelihood and descriptiveness databases are consulted to calculate scores for words and word pairs, then identification accuracy is improved, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-calculating descriptiveness scores and compound likelihood probabilities and storing them in databases before the actual similarity identification process. This allows the system to retrieve pre-computed values during identification rather than calculating them in real-time, significantly reducing processing time while maintaining high accuracy.
4Productivity
If rare words are assumed to be informative and common words to be uninformative, then the identification process is fast, but meaningful words may be ignored and misspellings may be incorrectly used for comparison
Solution Approach 1:
The patent changes the parameters for assessing word importance from simple frequency-based metrics to a multi-factor system that includes descriptiveness scores and compound likelihood probabilities. This allows the system to correctly identify meaningful words regardless of their frequency and to distinguish them from misspellings and non-informative terms.
Data Source
AI summary
The present invention provides systems and methods for identifying similar documents. In an embodiment, the present invention identifies similar documents by (1) receiving document text for a current document that includes at least one word; (2) calculating a prominence score and a descriptiveness score for each word and each pair of consecutive words; (3) calculating a comparison metric for the current document; (4) finding at least one potential document, where document text for each potential document includes at least one of the words; and (5) analyzing each potential document to identify at least one similar document.