Fixed Width Encoding for Search Posting Lists
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search engines face a tradeoff between time and space when implementing encoding schemes for posting lists, with simplified schemes requiring more memory but allowing quicker decoding, and complex schemes consuming less memory but having slower decoding processes.
Innovation Solution
A fixed width encoding scheme is implemented, where deltas in document identification numbers are recalculated as multiples and remainder values based on predetermined byte numbers, optimizing memory usage and decoding efficiency by determining the minimum number of bytes required to encode each delta.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a simplified encoding scheme is used, then decoding speed is improved, but memory space consumption increases
Solution Approach 1:
The patent applies dynamic encoding by selecting between different byte widths (1-byte or 2-byte) for each delta based on its value. This dynamic approach allows the system to optimize between space efficiency and decoding speed on a per-element basis, rather than using a static encoding scheme for the entire posting list.
Solution Approach 2:
The patent implements local quality by applying different encoding precision to different parts of the data. Small deltas (common in practice) are encoded in 1 byte for speed, while large deltas are encoded in 2 bytes for space efficiency. This localized differentiation resolves the contradiction by optimizing each region according to its specific characteristics.
2Quantity of substance
If a complex encoding scheme is used, then memory space consumption is reduced, but decoding speed decreases
Solution Approach 1:
The encoding scheme dynamically adjusts the number of bytes used for each delta based on its value. This dynamic adaptation allows the system to achieve space efficiency without uniformly sacrificing decoding speed, as the complexity is applied only where necessary (for large deltas).
Solution Approach 2:
The patent changes the parameter of byte width from a fixed value to a variable value that depends on the delta magnitude. This parameter change enables the system to switch between simple (1-byte) and complex (2-byte) encoding modes, resolving the contradiction between space efficiency and decoding speed.
3Quantity of substance
If variable byte encoding is used for deltas, then memory space is optimized, but decoding complexity increases
Solution Approach 1:
The patent segments the decoding process into two simple paths: checking if a delta value fits in 1 byte, and if not, using 2 bytes. This segmentation into discrete, manageable steps reduces decoding complexity compared to handling fully variable-length encodings, while still achieving space optimization.
Solution Approach 2:
By changing the byte width parameter based on delta value thresholds, the patent creates a controlled variable encoding scheme. This parameter change approach simplifies the decoding logic compared to fully adaptive schemes, as the parameter changes follow predictable patterns based on value ranges.
Data Source
AI summary
This specification describes technologies relating to fixed width encoding/decoding of document posting lists. In general, one aspect of the subject matter described in this specification can be embodied in apparatuses that include a server obtaining a list of one or more of document identification numbers, each of the document identification numbers uniquely identifying a document; an encoding device operatively connected to the server, the encoding device generating a sequence of deltas from the sequential list of one or more of the document identification numbers, and encoding each delta in the sequence of deltas using a fixed-width encoding scheme.