Bitmap Index Gap Encoding Without Max-Slot Padding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing bitmap index compression techniques inflate atom quantity and size due to the assumption of equal slots in data blocks, leading to inefficient storage and compression, especially with table compression and row deletion scenarios.
Innovation Solution
The proposed block-aware encoding method represents gaps using block-skip and slot-skip codes, allowing each data block to have a different number of bytes or bits, eliminating the need for padding and reducing the number of atoms required, and dynamically adjusts bit allocation to optimize compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If gap encoding is used to compress bitmaps in bitmap indexes, then compression ratio is improved, but atom quantity and size are inflated due to the assumption of equal slots in data blocks
Solution Approach 1:
The patent segments the bitmap into multiple blocks, where each block corresponds to a data block. Instead of treating the entire bitmap as a single continuous structure with uniform slot allocation, the invention divides it into independent segments that can have varying numbers of slots based on actual data block characteristics. This segmentation eliminates the need for padding atoms and reduces overall atom quantity while maintaining compression efficiency.
Solution Approach 2:
The patent introduces dynamic block-skip codes that can represent variable-length gaps between set bits across different blocks. Rather than using fixed-size atoms that assume equal slot distribution, the block-skip codes dynamically adapt to the actual number of slots in each data block, allowing the compression scheme to optimize atom representation based on real data characteristics rather than rigid assumptions.
2Device complexity
If max-slot factor assumption is used to simplify encoding, then encoding complexity is reduced, but compression efficiency deteriorates due to unnecessary padding
Solution Approach 1:
The patent applies local quality by allowing different blocks to have different slot configurations based on their actual content. Instead of imposing a uniform max-slot factor across the entire bitmap, each block can have its own slot count that matches the actual number of rows in the corresponding data block. This eliminates unnecessary padding in blocks with fewer slots while maintaining simple encoding within each local block context.
Solution Approach 2:
The patent changes the parameter representation from fixed max-slot factor to variable block-specific slot counts. By using block-skip codes that encode the actual number of slots in each block rather than assuming a maximum value, the system dynamically adjusts the encoding parameters to match real data characteristics, thereby improving compression efficiency without significantly increasing encoding complexity.
3Device complexity
If uniform slot allocation is assumed across data blocks, then bitmap structure is simplified, but storage efficiency decreases due to padding requirements
Solution Approach 1:
The patent segments the bitmap into multiple independent blocks, each corresponding to a data block. This segmentation allows each block to have its own slot allocation that matches the actual number of rows in the corresponding data block, eliminating the need for padding that would be required under uniform slot allocation. The segmented structure maintains simplicity within each block while achieving overall storage efficiency.
Solution Approach 2:
The patent introduces asymmetry by allowing different blocks to have different numbers of slots based on their actual content requirements. Instead of forcing a symmetric uniform slot allocation across all blocks, the system permits asymmetric slot distributions where each block adapts to its specific data characteristics, thereby eliminating wasted space from padding while maintaining manageable structure through the block-based organization.
Data Source
AI summary
Under block-aware encoding, a bitmap represented by atoms comprises a series of bitmaps for each data block in a database. Each bitmap in the series is referred to herein as a block bitmap. Each block bitmap may have a different number of bytes or bits. Gaps are represented in atoms using a pair of numbers referred to as a gap code. A gap code includes a block-skip code and slot-skip code. A block-skip code represents how many block bitmaps to advance to reach a subsequent block bitmap; a slot-skip code represents how many bytes to advance within the block bitmap to reach a byte with at least one bit set. A gap code is represented by bit positions within a byte, with some bit positions allocated to represent the block-skip code and some to represent the slot-skip code. The allocation is adjusted dynamically during encoding and decoding.