Text Deduplication via Substring Grouping and Inverted Index
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text deduplication methods are inefficient due to high time complexity when dealing with massive text files, as they require comparing multiple texts in pairs, leading to wasted resources and reduced deduplication efficiency.
Innovation Solution
A method and apparatus that capture subtext strings from each text, group texts with the same subtext strings into subsets, and perform deduplication processing on these subsets, reducing the need for pairwise comparisons by using inverted indexes and sliding window techniques to identify and remove duplicate texts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional pairwise comparison method is used for text deduplication, then deduplication accuracy can be maintained, but time complexity increases significantly and deduplication efficiency decreases
Solution Approach 1:
The patent segments the text comparison task by dividing texts into fixed-length substrings (e.g., 5-character substrings). Instead of comparing entire texts pairwise, the system compares substrings to identify potential duplicate groups. This segmentation reduces the comparison scope from O(n²) pairwise text comparisons to O(n×k) substring comparisons, where k is the number of substrings per text, significantly reducing time complexity while maintaining deduplication accuracy through subsequent verification steps.
Solution Approach 2:
The patent performs preliminary substring extraction and grouping before the actual deduplication process. By pre-processing texts to extract all fixed-length substrings and grouping texts that share common substrings, the system prepares candidate duplicate groups in advance. This preliminary action filters out non-duplicate texts early, reducing the number of texts that require detailed comparison and verification, thus reducing overall time complexity.
2Reliability
If pairwise comparison of all texts is performed, then complete deduplication can be achieved, but computational resources are wasted and processing speed decreases
Solution Approach 1:
The patent divides the large-scale text deduplication problem into smaller sub-problems by segmenting texts into substrings and grouping texts that share substrings. This segmentation allows parallel processing of different substring groups and reduces the computational burden on individual processing units, thereby improving processing speed while maintaining deduplication completeness through systematic verification.
Solution Approach 2:
The patent performs partial comparison by focusing only on texts that share common substrings rather than comparing all possible text pairs. This partial action approach processes only the necessary subset of text pairs that have potential duplication, avoiding wasted computational resources on obviously non-duplicate texts while ensuring complete deduplication of actual duplicates through verification steps.
3Productivity
If fixed-length substrings are used for text grouping, then deduplication efficiency improves through reduced comparisons, but the complexity of implementing substring extraction and management increases
Solution Approach 1:
The patent uses fixed-length substrings as a parameter change to simplify the deduplication process. By converting variable-length text comparison into fixed-length substring comparison, the system gains efficiency in grouping and comparison operations. The fixed length parameter enables efficient hashing, indexing, and grouping mechanisms, reducing system complexity in terms of comparison logic while improving deduplication efficiency through faster processing and memory management.
Data Source
AI summary
A text deduplication method and apparatus, and a storage medium are provided. The method includes: obtaining a text set, the text set including a plurality of to-be-deduplicated texts; capturing, for each to-be-deduplicated text, a corresponding subtext string from the to-be-deduplicated text; and determining, in the text set, to-be-deduplicated texts having a same subtext string, to obtain text subsets. Each subtext string corresponds to a text subset, and each text subset includes one or more to-be-deduplicated texts that have the corresponding subtext string. The method also includes performing text deduplication processing on the text subset corresponding to each subtext string, to obtain a deduplicated text set corresponding to each subtext string; and obtaining, according to the deduplicated text set corresponding to each subtext string, a result text set of the text set after the deduplication.