Feedback-Based Inverted Index Compression for Posting List Tuning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inverted indexes used for free-text search in large volumes of text face challenges in efficient storage and compression, leading to suboptimal performance due to fixed compression settings across all posting lists.
Innovation Solution
Implementing a feedback-based regeneration process for inverted indexes, where each posting list uses different compression settings, specifically using the Golomb-Rice2 compression method with individually settable K values, to optimize compression efficiency by periodically recalculating and updating compression settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If fixed compression settings are used across all posting lists, then device complexity is reduced, but compression efficiency deteriorates
Solution Approach 1:
The patent applies local quality by assigning different compression settings to different posting lists based on their individual characteristics. Each posting list can have its own compression parameters optimized for its specific data distribution and access patterns, rather than using a uniform compression setting across the entire inverted index. This localized optimization improves overall compression efficiency while maintaining manageable complexity through automated feedback-based adjustment.
Solution Approach 2:
The patent implements dynamics by making compression settings adjustable and changeable over time. The system periodically regenerates the inverted index and uses feedback from compression performance to dynamically update compression settings for individual posting lists. This allows the compression parameters to adapt to changing data characteristics and access patterns, improving compression efficiency without requiring permanent complexity in the system architecture.
2Loss of substance
If compression settings are individually optimized for each posting list, then compression efficiency is improved, but device complexity increases
Solution Approach 1:
The patent applies feedback by implementing a regeneration process that monitors compression performance and uses this information to adjust compression settings for individual posting lists. The system periodically evaluates the effectiveness of current compression settings and automatically updates them based on observed performance metrics, eliminating the need for manual configuration and reducing the perceived complexity while maintaining optimized compression.
Solution Approach 2:
The patent implements self-service by enabling the inverted index system to automatically optimize its own compression settings without external intervention. The feedback-based regeneration process allows the system to self-adjust compression parameters for each posting list based on its specific characteristics and performance requirements, reducing the operational complexity of managing individualized compression settings.
3Loss of substance
If periodic regeneration with decompression is performed, then compression efficiency is improved over time, but processing time increases
Solution Approach 1:
The patent applies periodic action by scheduling inverted index regeneration at intervals rather than continuously. This allows the system to balance compression optimization with processing overhead by performing decompression and re-compression operations only when necessary to update compression settings, rather than continuously. The periodic regeneration maintains compression efficiency improvements while limiting the time penalty to discrete intervals.
Data Source
AI summary
The disclosed technology is generally directed to the compression of inverted indexes. In one example of the technology, an inverted index that includes a plurality of posting lists and metadata is provided. The inverted index indicates compression settings that are associated with the plurality of posting lists. At periodic scheduled times, a regeneration is performed on the inverted index. The regeneration includes decompressing the inverted index. The decompressing uses the compression settings indicated by the inverted index. The regeneration further includes determining compression settings to use during a next periodic scheduled time of the plurality of periodic scheduled times, such that at least a first posting list of the plurality of posting lists uses a different compression setting than a second posting list of the plurality of posting lists.