Textual Near-Duplicate Grouping via Non-Contiguous Character Sequence Analysis

The patent segments the document comparison process into multiple stages: first using fast filtering techniques (character n-gram counts, word frequency comparisons) to eliminate clearly dissimilar documents, then applying more computationally intensive word-by-word comparison only to documents that pass the filtering stage. This hierarchical segmentation resolves the contradiction by achieving high precision through detailed comparison only when necessary, while maintaining fast initial screening for the majority of document pairs.

The patent introduces intermediate filtering techniques that act as mediators between fast heuristic methods and accurate word-by-word comparison. These intermediate filters (such as character n-gram overlap analysis and weighted word frequency comparisons) provide a middle ground that significantly reduces false positives compared to simple heuristic methods, while still being much faster than complete word-by-word comparison. This intermediary layer resolves the contradiction by maintaining high processing speed while substantially improving reliability.

The patent segments the computational workload into hierarchical stages with increasing resource requirements. The majority of document pairs are processed using low-resource filtering techniques that consume minimal energy. Only a small fraction of document pairs that pass through multiple filtering stages are subjected to high-resource word-by-word comparison. This segmentation resolves the energy precision contradiction by allocating extensive processing resources only to the minority of cases where high precision is actually needed, while using minimal resources for the majority of comparisons.