Near Duplicate Content Detection Using Tokenization and Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In online commerce and networked systems, identifying near duplicate user-generated content is challenging due to the large number of listings, rapidly changing marketplace, and the presence of duplicate or near-duplicate listings intended to maximize exposure without additional costs.
Innovation Solution
A computer-implemented system and method that automatically identifies near duplicate user-generated content by tokenizing listings, removing un-descriptive words, creating sets of portions, assigning weights, calculating document magnitudes, and using similarity measures like Cosine Measure or Tanimoto Coefficient to detect resemblance, while also linking accounts associated with near duplicate content to determine if they belong to the same user.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If sellers post multiple duplicate or near-duplicate listings to maximize exposure, then the number of listings increases, but the quality and uniqueness of content decreases
Solution Approach 1:
The patent segments listings into distinct groups by identifying near-duplicates through tokenization and similarity comparison. This segmentation separates unique content from duplicate content, allowing the system to manage listings in meaningful groups rather than treating all listings equally.
Solution Approach 2:
The patent introduces an intermediary detection system that acts as a mediator between sellers and the listing database. This intermediary automatically identifies near-duplicate listings before they are published, preventing duplicate content from entering the system in the first place.
2Measurement precision
If the system performs comprehensive duplicate detection on all listings, then the accuracy of duplicate identification improves, but the processing time and computational resources increase
Solution Approach 1:
The patent applies partial action by using cosine similarity thresholds to identify only sufficiently similar listings for further review. Rather than performing exhaustive analysis on all possible listing pairs, the system focuses computational resources on comparing listings that exceed a certain similarity threshold, achieving good enough detection without excessive processing.
Solution Approach 2:
The patent changes parameters by using different similarity thresholds and weighting schemes for different types of content. The system can adjust the sensitivity of duplicate detection by modifying these parameters, allowing it to balance between detection accuracy and processing speed based on specific needs.
3Ease of operation
If sellers use duplicate listings to gain greater exposure, then the visibility of products increases, but the integrity of the marketplace decreases
Solution Approach 1:
The patent implements feedback mechanisms where the system monitors listing patterns and provides signals to sellers when duplicate content is detected. This feedback loop allows sellers to understand when their listings are being flagged and adjust their behavior accordingly, maintaining marketplace integrity while still allowing legitimate multiple listings.
Data Source
AI summary
A computer-implemented system and method for identification of near duplicate user-generated content in a networked system are disclosed. The apparatus in an example embodiment includes a data receiver to receive a first instance of user-generated content; a tokenizer to tokenize the first instance into a set of words, create a set of portions from the tokenized first instance, and assign weight to each portion of the set of portions; a magnitude calculator to calculate a magnitude for the first instance based on the weight of each portion; a resemblance score calculator to search a data store for a second instance with at least one portion in common with the first instance and calculate a resemblance score between the first instance and the second instance; and an account linker to link accounts associated with each of the first instance and the second instance.