Inversion Buffer for Index Generation Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for generating inverted indexes are resource-intensive and time-consuming, particularly due to the reliance on intermediate files, which slows down the creation of indexes and can make them less inclusive and current.
Innovation Solution
The method involves allocating an inversion buffer in memory to efficiently order data sets of tuples by reserving space for keys and their associated references, allowing for the direct insertion and ordering of postings, thereby eliminating the need for multiple intermediate files and reducing reliance on disk operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional inversion methods using intermediate files are used, then the index can be generated, but the process is time-consuming and resource-intensive
Solution Approach 1:
The patent extracts the sorting operation from the file system and moves it into memory by allocating an inversion buffer that directly stores postings in sorted order. This eliminates the need to write intermediate files to disk and read them back for sorting, thereby removing the time-consuming I/O operations while maintaining the indexing functionality.
Solution Approach 2:
The inversion buffer acts as an intermediary data structure in memory that receives postings and automatically maintains them in sorted order using a buffer manager. This mediator eliminates the need for traditional intermediate files on disk, replacing them with an in-memory structure that provides both sorting and storage functions simultaneously.
2Manufacturing precision
If multiple intermediate files are created and merged, then all postings can be sorted, but disk operations slow down the process
Solution Approach 1:
The patent replaces the mechanical file system operations (writing intermediate files to disk, reading them back, and merging them) with an in-memory buffer management system. The buffer manager maintains postings in sorted order using memory operations, which are significantly faster than disk operations, thereby substituting the mechanical I/O system with a software-based memory management system.
Solution Approach 2:
The inversion buffer is segmented into multiple regions or slots that can independently store and manage postings. This segmentation allows the buffer to handle large numbers of postings efficiently by distributing them across multiple memory regions, maintaining sorted order within each segment while enabling parallel processing and reducing contention.
3Quantity of substance
If intermediate files are used for sorting, then memory constraints are relaxed, but the overall process becomes slower
Solution Approach 1:
The patent changes the dimension of data storage from the file system (disk) to the memory system (RAM). By allocating the inversion buffer in memory rather than using intermediate files on disk, the system exploits the speed dimension of memory operations, achieving orders of magnitude improvement in access time while maintaining the ability to handle large quantities of postings through buffer management techniques.
Data Source
AI summary
Methods and systems for the generation of computer readable indexes or other ordered lists are provided. A corpus of electronic documents or other electronic information is parsed into postings that include key and reference pairs. An inversion buffer in memory is explicitly or implicitly formatted to receive the postings in a predetermined order by key. Each key is assigned a space in the inversion buffer that is subsequently filled with references associated with the key during an inversion method. In an embodiment, an index file is generated directly from the inversion buffer, or in the case of large inversions, from a plurality of inversion buffer segments.