Bigram Bitmap Duplicate Detection in Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional algorithms are not scalable for handling large volumes of diverse data and require prior standardization to detect potential duplicates, which limits their effectiveness in identifying similar values with spelling errors or different orders.
Innovation Solution
The method converts attribute values into bigrams, represents them as bits in a bitmap, and uses bitwise operations to group similar values, eliminating the need for prior standardization and enabling efficient detection of potential duplicates without prior knowledge of the data domain.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional algorithms are used for finding duplicates, then prior standardization can be performed to ensure accuracy, but scalability is reduced for handling large volumes of diverse data
Solution Approach 1:
The patent segments attribute values into bigrams (character pairs) and represents them as individual bits in a bitmap. This segmentation transforms the complex problem of comparing entire attribute values into a simpler problem of comparing binary bitmaps, enabling scalable processing of large volumes of diverse data while maintaining the ability to detect potential duplicates accurately
Solution Approach 2:
The patent changes the representation parameter of attribute values from their original form to a binary bitmap representation. By converting attribute values into bitmaps where each bit corresponds to a bigram, the system enables efficient bitwise operations that scale well with data volume while preserving the information needed for accurate duplicate detection
2Reliability
If traditional algorithms require prior standardization, then data quality can be improved, but processing time and complexity increase
Solution Approach 1:
The patent performs preliminary conversion of attribute values into bitmap representations based on bigrams before the actual duplicate detection process. This preliminary action creates a standardized form that facilitates efficient comparison while reducing the need for extensive prior standardization, thereby improving data quality for duplicate detection without significantly increasing processing time
Solution Approach 2:
The patent substitutes traditional mechanical string comparison and standardization processes with bitmap-based bitwise operations. This substitution replaces complex mechanical processing of attribute values with efficient binary operations, reducing processing time while maintaining the ability to identify potential duplicates with high reliability
3Measurement precision
If traditional algorithms are used, then exact duplicates can be identified, but potential duplicates with spelling errors or different orders are missed
Solution Approach 1:
The patent segments attribute values into overlapping bigrams that capture local patterns within the data. This segmentation approach allows the system to identify potential duplicates with spelling errors or different word orders by detecting significant overlaps in bigram patterns, thereby increasing adaptability to diverse data varieties while maintaining precision for exact duplicates
Solution Approach 2:
The patent changes the detection parameter from exact string matching to bigram-based similarity measurement using bitmap operations. This parameter change enables the system to detect potential duplicates with spelling errors or different orders by identifying significant overlaps in their bigram representations, thereby increasing versatility without sacrificing the ability to identify exact duplicates
4Measurement precision
If prior sorting or classification is performed, then duplicate detection accuracy improves, but processing overhead increases
Solution Approach 1:
The patent performs preliminary conversion to bitmap representation as a lightweight preprocessing step that does not require extensive sorting or classification. This preliminary action creates a standardized form suitable for efficient comparison while minimizing processing overhead, thereby maintaining duplicate detection accuracy without significantly increasing device complexity
Solution Approach 2:
The patent substitutes complex sorting and classification mechanisms with a simpler bitmap-based comparison approach. By replacing mechanical sorting processes with efficient bitmap operations, the system reduces processing overhead while maintaining the ability to accurately identify potential duplicates through pattern matching in the bitmap representations
Data Source
AI summary
A method, system and computer program product for finding groups of potential duplicates in attribute values. Each attribute value of the attribute values is converted to a respective set of bigrams. All bigrams present in the attribute values may be determined. Bigrams present in the attribute values may be represented as bits. This may result in a bitmap representing the presence of the bigrams in the attribute values. The attribute values may be grouped using bitwise operations on the bitmap, where each group includes attribute values that are determined based on pairwise bigram-based similarity scores. The pairwise bigram-based similarity score reflects the number of common bigrams between two attribute values.