Dual-lexicon duplicate document detection precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing duplicate detection systems, such as single-signature techniques, face precision issues when dealing with spam emails that have varying content, leading to false positives and increased resource consumption for e-mail service providers.
Innovation Solution
The implementation of a dual-lexicon approach, where a primary lexicon and a secondary lexicon of attributes are used to generate a signature for documents, with the secondary lexicon supplementing the primary lexicon when the intersection of unique attributes falls below a threshold, ensuring the signature exceeds the threshold and improves precision in identifying duplicate documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single-signature duplicate detection is used, then device complexity is reduced, but measurement precision deteriorates leading to false positives
Solution Approach 1:
The lexicon is segmented into two distinct parts: a primary lexicon containing common words and a secondary lexicon containing unique words. This segmentation allows the system to handle different types of words separately, improving detection precision without requiring a complete redesign of the entire detection system. The primary lexicon handles frequent words that appear in many documents, while the secondary lexicon captures distinctive words that differentiate duplicate documents.
Solution Approach 2:
The patent combines two different lexicon types (primary and secondary) into a composite detection approach. By merging the results from both lexicons and applying a threshold-based filtering mechanism, the system achieves higher measurement precision while maintaining manageable complexity through the structured combination of simpler components.
2Device complexity
If a single lexicon is used, then device complexity is low, but reliability deteriorates due to false positives from varying spam content
Solution Approach 1:
The lexicon is divided into a primary lexicon for common words and a secondary lexicon for unique words. This segmentation enables the system to reliably detect duplicates even when spam content varies, because the secondary lexicon captures distinctive words that remain consistent across duplicate spam emails while the primary lexicon handles common variations.
Solution Approach 2:
The patent changes the parameter of lexicon composition by using two different lexicons with different characteristics rather than a single lexicon. The primary lexicon focuses on common words with lower specificity, while the secondary lexicon focuses on unique words with higher specificity. This parameter change improves reliability by allowing the system to adapt to varying spam content while maintaining consistent duplicate detection.
3Measurement precision
If more attributes are intersected to improve precision, then measurement precision improves, but loss of time increases due to additional processing
Solution Approach 1:
The attribute intersection process is segmented into two stages corresponding to the two lexicons. The first stage intersects with the primary lexicon (faster, fewer attributes), and the second stage intersects with the secondary lexicon (more precise, additional attributes). This segmentation allows the system to achieve high precision without processing all attributes uniformly, thereby reducing overall processing time.
Solution Approach 2:
The patent applies partial action by using the primary lexicon intersection as a preliminary filter. When this initial intersection already provides sufficient precision, the system can stop early without performing the more time-consuming secondary lexicon intersection. This partial action approach achieves the necessary precision for many cases while avoiding unnecessary processing time.
Data Source
AI summary
In a single-signature duplicate document system, a secondary set of attributes is used in addition to a primary set of attributes so as to improve the precision of the system. When the projection of a document onto the primary set of attributes is below a threshold, then a secondary set of attributes is used to supplement the primary lexicon so that the projection is above the threshold.