Inverted Index Construction Using Heap Data Structures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The creation of an inverted index from data streams is resource-intensive, consuming significant processing power and memory storage due to the need for large-scale indexing and hashing of documents.
Innovation Solution
A method involving dividing documents into n-grams, hashing them, sorting and deduplicating the hashed n-grams, and using a heap data structure to efficiently build the inverted index, which limits memory usage to the number of documents rather than the number of hashes, and allows for streaming data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional inverted index construction methods are used, then the index can be built, but memory consumption and processing power requirements become excessively high
Solution Approach 1:
The patent divides documents into n-grams (segments of tokens) and processes them in sorted order, grouping by hashed n-gram values. This segmentation allows the system to handle large datasets in manageable chunks, loading only necessary portions into memory at any given time rather than requiring all data to be loaded simultaneously.
Solution Approach 2:
The patent performs preliminary sorting of hashed n-grams before the main indexing process. By pre-sorting the data and organizing it by hashed value, the system eliminates the need for complex in-memory hashing and collision resolution during index construction, significantly reducing peak memory requirements while maintaining indexing reliability.
2Reliability
If all n-grams are loaded into memory for indexing, then complete indexing can be achieved, but memory usage scales with the number of hashes rather than documents
Solution Approach 1:
The patent segments the indexing process into external sorting followed by sequential processing. By sorting n-grams on disk and then processing them in sorted order, the system maintains indexing completeness while limiting memory usage to storing only the current batch of n-grams being processed, not all n-grams simultaneously.
Solution Approach 2:
The patent introduces an external sorting mechanism as an intermediary between data loading and index construction. This intermediary step organizes data on disk before processing, allowing the main indexing algorithm to work with small, manageable in-memory batches while maintaining access to the complete dataset through the sorted external structure.
3Measurement precision
If traditional hashing methods are used for all n-grams, then accurate document mapping is achieved, but processing time and computational complexity increase significantly
Solution Approach 1:
The patent performs preliminary sorting of hashed n-grams by their hash values before the main indexing process. This pre-sorting eliminates the need for repeated hashing and collision resolution during index construction, reducing processing time while maintaining the accuracy of document mapping through the preserved hash-based organization.
Solution Approach 2:
The patent replaces the mechanical hashing-and-collision-resolution process with a sorting-based approach. By substituting the hash table insertion mechanism with external sorting followed by sequential processing, the system maintains precise document mapping through sorted order preservation while dramatically reducing computational complexity and processing time.
Data Source
AI summary
Systems and methods create an inverted index for a set of documents utilizing a heap data structure. The documents can be divided into a series of n-grams. The n-grams can be hashed, producing hashed n-gram values. A first hashed n-gram value can be placed into the heap data structure. Operations performed using the heap data structure obtain document identifiers that are associated with documents that include a hashed n-gram value that matches the top n-gram value in the heap.