Integer Fingerprint Duplicate Checking for Large-Scale Text Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text similarity calculation methods are inefficient for large-scale text analysis and duplicate checking, particularly when dealing with long texts, due to high storage requirements and slow query times.
Innovation Solution
The method involves creating and storing fingerprints in integer form using byte data, and employing an integer XOR operation to quickly calculate the distance between fingerprints, thereby improving the efficiency of similarity calculation and duplicate checking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If string data is used to store document fingerprints, then the fingerprints can be stored and compared, but a large amount of space is occupied and query time increases
Solution Approach 1:
The patent changes the data type parameter of fingerprint storage from string format to integer format. This parameter change reduces the storage space required for each fingerprint from multiple bytes (string) to 4 bytes (integer), and enables faster comparison operations through integer-based Hamming distance calculation instead of string manipulation
Solution Approach 2:
The patent replaces the mechanical string comparison process with an integer-based computational system. Instead of comparing character-by-character strings, the system uses integer arithmetic operations (XOR, bit counting) to calculate Hamming distance, which is computationally more efficient and faster
2Productivity
If traditional similarity calculation methods are used for large-scale texts, then accuracy can be maintained, but the calculation efficiency is low
Solution Approach 1:
The patent extracts the essential similarity calculation into a simplified integer-based Hamming distance metric. By representing documents as integer fingerprints and comparing only these compact representations, the system maintains the core similarity measurement function while dramatically reducing computational complexity for large-scale text comparison
Solution Approach 2:
The patent creates a simplified copy of the document representation in the form of integer fingerprints. Instead of working with the full text or complex string representations, the system uses these integer copies for comparison, which preserves the essential similarity information while enabling faster processing
Data Source
AI summary
Provided are a text duplicate checking method, an electronic device and a computer-readable storage medium. The method includes storing a fingerprint set and a corresponding text ID in a byte data manner to obtain a fingerprint library; acquiring a target text and creating a target fingerprint; obtaining a comparison fingerprint set from map memories according to the target fingerprint, and calculating a similarity between the target fingerprint and each comparison fingerprint in the comparison fingerprint set separately; and based on a determination result that a number of 1s in binary values of one similarity is less than or equal to a preset value, querying a text ID corresponding to the one similarity, to implement duplicate checking of the target text.