Inverted Index Position Vectors for Fast Compressed Lookup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for inverted indices in computer science often face trade-offs between compression ratio and decompression speed, with techniques like Golomb coding optimizing for one aspect at the expense of the other, and lack efficient methods for representing location information in datasets with 1:1 and 1:n value mappings.
Innovation Solution
The proposed solution involves a computer-implemented method using a position vector with indicator bits to differentiate between 1:1 and 1:n values, employing variable byte-length encoding, variable block-length coding, and optimized unary prefix encoding to achieve faster access speeds and smaller index sizes, while eliminating the need for count information typically required in Golomb coding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If Golomb coding is used for compression, then compression ratio is improved, but decompression speed deteriorates
Solution Approach 1:
The patent segments the inverted index into two distinct parts: a position vector containing indicator bits for 1:1 values, and separate location lists for 1:n values. This segmentation allows different compression strategies to be applied to different data types, optimizing both compression ratio and access speed by treating frequently accessed 1:1 values differently from multi-location 1:n values.
Solution Approach 2:
The patent applies different encoding qualities to different parts of the data structure. The position vector uses compact indicator bits for 1:1 values enabling fast access, while location lists use compression for 1:n values. This local differentiation optimizes the overall system by matching the encoding strategy to the access patterns and characteristics of each data segment.
2Loss of information
If traditional inverted index structures are used, then completeness of location information is improved, but index size increases
Solution Approach 1:
The patent extracts the indicator bits from the traditional uniform encoding structure and places them in a separate position vector. This extraction allows the system to identify and separately handle 1:1 values, storing only their positions in the position vector rather than full location lists, thereby reducing overall index size while preserving complete location information through the combination of position vector and location lists.
Solution Approach 2:
Instead of storing complete location lists for all values as in traditional inverted indexes, the patent inverts the approach by storing only position indicators for 1:1 values and using compact references for 1:n values. This inversion reduces redundancy and minimizes index size while maintaining the ability to retrieve complete location information when needed.
3Speed
If variable byte-length encoding is used, then decompression speed is improved, but manufacturing complexity increases
Solution Approach 1:
The patent performs preliminary classification of values into 1:1 and 1:n categories during the indexing phase, setting indicator bits in advance. This preliminary action enables faster decompression because the structure of the data is predetermined, allowing the decompression algorithm to quickly determine whether to read a position directly or to retrieve a location list, without requiring complex runtime analysis.
Data Source
AI summary
Methods and systems for providing an inverted index for a dataset are disclosed. The inverted index includes a position vector, with fields that correspond to values in the indexed dataset. The fields include data to be used in determining where each value appears in the dataset. The position vector is populated differently for different value types. A 1:1 value appears once in the dataset; a 1:n value appears multiple times. For a 1:1 value, the position vector stores information for where that value appears. For a 1:n value, the position vector stores a pointer, e.g. a memory reference, that identifies a list of locations where the value appears. The list can be encoded or otherwise compressed. A set of indicators can be stored for the fields indicating whether the field has 1:n or 1:1 value information. The indicator is used to control interpretation of the information in a field.