Deep Learning Document Similarity for Accurate Duplicate Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for measuring document similarity using mathematical measures often overestimate or underestimate semantic similarity due to accidental matches or mismatches in unimportant parts, leading to inaccurate duplicate document detection.
Innovation Solution
A deep learning-based similarity model is trained using nonlinear functions to adjust mathematical similarities based on document attributes, enhancing the accuracy of semantic similarity calculations by compensating for underestimation and overestimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If mathematical measures (cosine similarity, Euclidean distance, Jaccard similarity) are used to measure document similarity, then the similarity calculation is simple and fast, but the measurement precision is poor due to overestimation or underestimation of semantic similarity
Solution Approach 1:
The patent transforms the similarity measurement from direct mathematical comparison to a learned parameter space. A deep learning model (such as BERT) processes document pairs and outputs a similarity score between 0 and 1, where the model parameters are trained to capture semantic relationships. This changes the measurement from static mathematical formulas to dynamic learned parameters that adapt to semantic contexts, resolving the contradiction between simplicity and accuracy.
2Measurement precision
If deep learning models are used to improve similarity measurement accuracy, then the measurement precision improves, but the resource consumption and computational complexity increase
Solution Approach 1:
The patent performs preliminary actions by pre-training deep learning models on large corpora before actual duplicate detection. The model learns semantic representations in advance, so during inference, it only needs to compute similarity scores for document pairs rather than performing heavy training computations. This preliminary learning phase separates the high computational cost from the actual detection task, reducing real-time resource consumption while maintaining high accuracy.
3Reliability
If conventional similarity measurement methods are used, then the processing speed is fast, but the reliability of duplicate detection is low due to false positives and false negatives
Solution Approach 1:
The patent replaces mechanical similarity computation (direct mathematical comparison of text tokens) with a neural network-based semantic understanding system. Instead of mechanically comparing word overlaps using cosine similarity or Jaccard index, the system uses trained neural networks to understand semantic meaning, context, and paraphrasing. This substitution dramatically improves detection reliability by capturing semantic equivalence that mechanical methods miss, while the trained model's efficient inference maintains acceptable processing speeds.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed is a method and system, the method including extracting similar and dissimilar document pair sets from a document database, the similar document pair set including similar document pairs having a common attribute, and the dissimilar document pair set including dissimilar document pairs extracted randomly, calculating a mathematical similarity for each of the similar and dissimilar document pairs using a mathematical measure to obtain a first and second mathematical similarities, calculating a semantic similarity for each of the similar and dissimilar document pairs to obtain a first and second semantic similarities, the first semantic similarities being higher than the first mathematical similarities, and the second semantic similarities being lower than the second mathematical similarities, training a similarity model based on the similar and dissimilar document pairs, and the first and second semantic similarities to obtain a trained similarity model, and detecting a duplicate document using the trained similarity model.