Compressed Bitmap Index for Iceberg Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in computing iceberg queries due to tuple-scan based approaches, which require significant CPU time and disk access, especially when dealing with large tables and small result sets, as they need to read and scan all tuples to obtain accurate aggregate results.
Innovation Solution
Implementing a bitmap index with dynamic pruning and vector alignment techniques to process iceberg queries, allowing for efficient computation by eliminating non-meeting bitmap vectors and aligning relevant vectors to avoid unnecessary bitwise operations, thereby reducing disk access and computation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If tuple-scan based approaches are used to compute iceberg queries, then accurate aggregate results can be obtained, but significant CPU time and disk access are required
Solution Approach 1:
The patent pre-computes and stores bitmap indexes for each attribute before query execution. These bitmaps capture the presence/absence of attribute values across tuples, allowing the query processor to perform aggregate computations using bitwise operations on pre-prepared data structures rather than scanning raw tuples during query time.
Solution Approach 2:
The patent replaces the mechanical tuple-by-tuple scanning process with bitwise operations on bitmap vectors. Instead of reading and processing individual tuples sequentially, the system uses efficient bitwise AND, OR, and count operations on compressed bitmap representations, leveraging bit-level parallelism to accelerate aggregate computations.
2Loss of information
If all tuples are read and scanned to obtain accurate aggregate results, then complete information is processed, but performance becomes unsatisfactory for large tables with small result sets
Solution Approach 1:
The patent extracts only the essential presence/absence information from tuples and stores it in compact bitmap indexes. During query processing, it extracts and processes only the relevant bitmap vectors corresponding to attributes in the GROUP BY clause, avoiding the need to read and process entire tuples or all attributes, thus filtering out unnecessary data access.
Solution Approach 2:
The patent changes the representation parameter of data from full tuple records to compressed bitmap vectors. This parameter transformation allows the system to work with much more compact data structures that retain the essential information needed for aggregate computations while dramatically reducing memory and disk access requirements.
3Loss of time
If bitmap indexes are used to process iceberg queries, then disk access and computation time are reduced, but additional index maintenance overhead is introduced
Solution Approach 1:
The patent designs the bitmap index structure to serve multiple purposes: it supports exact match queries, range queries, and aggregate computations (COUNT, SUM, AVG) on grouped data. The same bitmap vectors that enable efficient filtering also provide the basis for all aggregate operations, eliminating the need for separate index structures for different query types.
Solution Approach 2:
The patent segments the bitmap index into separate vectors for each attribute, allowing independent processing and optimization of each attribute's data. This segmentation enables the system to load and process only the specific attribute bitmaps needed for a given query, rather than maintaining and accessing a monolithic index structure.
Data Source
AI summary
Exemplary embodiments include an iceberg query method, including processing the iceberg query using a bitmap index having a plurality of bitmap vectors in a database, eliminating any of the plurality of bitmap vectors in the bitmap index that fails to meet a given condition thereby forming a subset of the plurality of bitmap vectors and aligning the vectors in the subset of the plurality of bitmap vectors in the bitmap index according to respective positions of the bitmap vectors in the subset of the plurality of bitmap vectors.