Inverted Index Segmentation for Search Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engines face inefficiencies due to the heavy burden on Central Processing Unit and magnetic disks caused by large inverted index data, resulting from mixing high and low frequency term indexes in a single device, which leads to slow access speeds and high memory occupancy.
Innovation Solution
Separate high frequency and low frequency term indexes and store them in different retrieval modules corresponding to independent storage devices, optimizing data access and reducing read/write operations on magnetic disks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If inverted indexes for all terms are stored in a single computing device, then the search engine can maintain a unified index structure, but the storage space occupancy increases and the access speed decreases due to heavy read/write operations on magnetic disks
Solution Approach 1:
The patent divides the inverted index into multiple segments based on term frequency: high-frequency term inverted indexes and low-frequency term inverted indexes. Each segment is stored in separate storage devices, reducing the burden on any single device and improving access speed while maintaining the overall index structure.
2Device complexity
If inverted indexes for all terms are stored in a single computing device, then the index management is simplified, but the memory occupancy increases and the computation burden on CPU and magnetic disks becomes too heavy
Solution Approach 1:
The patent segments the inverted index data into high-frequency and low-frequency parts, storing them in different storage devices. This segmentation reduces memory occupancy in any single device and distributes the computation burden, while the management complexity increases only slightly due to the clear segmentation strategy.
3Adaptability or versatility
If the inverted index data is large in volume, then the search engine can index more terms from 200 million documents (1.4 billion terms), but the read and write operations on magnetic disks increase, resulting in low efficiency
Solution Approach 1:
The patent segments the large volume of inverted index data into high-frequency and low-frequency segments, storing them in separate storage devices. This allows the system to maintain comprehensive term coverage (1.4 billion terms from 200 million documents) while reducing the read/write operation burden on any single magnetic disk, thereby improving search efficiency.
4Device complexity
If high frequency terms and low frequency terms are mixed in a single inverted index, then the index structure is unified, but the read operations on magnetic disks cannot be optimized for different access patterns
Solution Approach 1:
The patent segments the inverted index into high-frequency and low-frequency parts with different access patterns. High-frequency terms are stored in storage devices optimized for fast access, while low-frequency terms are stored in other storage devices. This segmentation optimizes data access for different term types while maintaining relatively simple index structures.
Data Source
AI summary
Methods and apparatuses for search are provided and related to the field of search technology. A method may include: performing term segmentation for grabbed documents to count a term frequency of each term, the term frequency of the term representing a number of the grabbed documents containing the term; generating a high frequency term inverted index and a low frequency term inverted index respectively, wherein the high frequency term inverted index contains terms having a term frequency higher than a predefined threshold, and the low frequency term inverted index contains terms having a term frequency not higher than the predefined threshold; and loading the high frequency term inverted index and the low frequency term inverted index respectively to different retrieval modules, the different retrieval modules respectively corresponding to mutually independent storage devices.