Bitset Optimized Inverted Index for Duplicate Document Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional mechanisms for identifying duplicate documents in large document corpora, such as those containing millions or hundreds of millions of documents, are inefficient and lack precision.
Innovation Solution
The implementation of a bitset optimized inverted index, utilizing adaptive tokenization and bitsets to efficiently identify candidate duplicate documents by tokenizing documents, creating a data structure that associates tokens with documents, and compressing bitsets for memory efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional term frequency-based mechanisms are used to identify duplicate documents, then the approach is simple to implement, but the precision and effectiveness deteriorate when dealing with document corpora containing millions or hundreds of millions of documents
Solution Approach 1:
The patent segments the document corpus into manageable units by creating an inverted index that maps tokens to document identifiers. This segmentation allows the system to process large corpora by breaking down the global duplicate detection problem into localized token-based comparisons, thereby improving precision without requiring monolithic complex structures.
Solution Approach 2:
The patent transitions from traditional term frequency analysis to a bitset-based dimensional representation where each token is associated with a bitset of document identifiers. This dimensional change enables efficient set operations (intersection, union, difference) on document sets, dramatically improving duplicate identification precision while maintaining manageable computational complexity through bitwise operations.
2Measurement precision
If a comprehensive inverted index is created to improve duplicate identification accuracy, then the measurement precision improves, but the memory consumption increases significantly
Solution Approach 1:
The patent creates compact bitset copies of document identifier sets for each token in the inverted index. Instead of storing full document metadata or complex data structures, it uses minimal bitset representations that can be efficiently copied and manipulated in memory. This copying approach maintains high duplicate identification precision while dramatically reducing memory consumption compared to traditional inverted indexes.
Solution Approach 2:
The patent changes the parameter representation from traditional term frequency counts to binary bitset indicators of document membership. This parameter transformation allows the system to achieve high measurement precision through set operations while optimizing memory usage by storing only essential presence/absence information in compact bitset format rather than elaborate data structures.
3Quantity of substance
If bitset optimization is applied to reduce memory usage, then memory efficiency improves, but the processing speed may deteriorate due to additional compression and decompression operations
Solution Approach 1:
The patent performs preliminary bitset compression during index construction, pre-processing the data into an optimized storage format. By compressing bitsets in advance and organizing the inverted index structure beforehand, the system reduces memory usage while minimizing the impact on processing speed during duplicate detection queries, as the compressed structure is already prepared for efficient access.
4Measurement precision
If adaptive tokenization is implemented to improve duplicate detection accuracy, then the measurement precision improves, but the device complexity and processing overhead increase
Solution Approach 1:
The patent implements adaptive tokenization that dynamically selects or adjusts tokenization strategies based on the specific characteristics of the document corpus and query patterns. This dynamic approach improves duplicate detection accuracy by adapting to different data scenarios while managing complexity through conditional logic rather than fixed complex structures, allowing the system to optimize performance for specific use cases.
Data Source
AI summary
Technologies are disclosed for mapping documents to candidate duplicate documents in a document corpus. A bitset optimized inverted index is created for a document corpus. A document is received for which candidate duplicate documents in the document corpus are to be identified. The document is tokenized using adaptive tokenization. A determination made as to whether tokens in the document are represented in the bitset optimized inverted index. A list of candidate duplicate documents is created for tokens represented in the optimized inverted index utilizing in-memory bitsets that map tokens to documents that contain the tokens in the document corpus.