Bitmap Index Iterative Filtering for Database Query Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database query methods, particularly for iceberg queries, are inefficient due to high computational complexity and space requirements of bitmap indices, especially when dealing with large databases and high attribute cardinalities, leading to unsatisfactory performance in retrieving aggregate values.
Innovation Solution
The method involves creating a filter bitmap to iteratively process matching entries and subtracting filtered group bitmaps to reduce the number of necessary iterations, using available information in both bitmap and column store spaces, thereby minimizing the number of required bitmap combinations and improving query efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a bitmap index is used to represent sets of rows in a database, then query efficiency for exact value matching is improved, but the space required for storing bitmap indices increases significantly when attribute cardinalities are high
Solution Approach 1:
The patent applies segmentation by dividing the bitmap index into multiple segments or partitions based on attribute values. Instead of creating a single comprehensive bitmap for all possible values, the index is segmented into multiple smaller bitmaps, each handling a subset of values. This reduces the memory footprint of individual bitmap structures while maintaining the ability to efficiently query any value by accessing only the relevant segment.
Solution Approach 2:
The patent introduces a hierarchical dimension to the bitmap index structure by organizing bitmaps in multiple levels. The first level contains bitmaps for high-cardinality attributes with selective indexing, while subsequent levels provide more granular indexing for specific value ranges. This dimensional organization allows the system to balance between query efficiency and space consumption by only loading and maintaining bitmaps for the most frequently queried value ranges.
2Adaptability or versatility
If a bitmap index with many bitmaps is used to handle high attribute cardinalities, then coverage of more attribute values is achieved, but processing time for range queries increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing aggregated statistics about the bitmap data during index construction. Metadata such as the number of set bits, value ranges, and frequency distributions are calculated in advance and stored with the bitmap structure. During range queries, this pre-computed information allows the system to quickly determine which bitmaps are relevant and skip unnecessary processing, significantly reducing query time while maintaining comprehensive value coverage.
3Measurement precision
If tuple-scan based approaches are used to compute aggregate functions, then all tuples can be processed to ensure accurate results, but the CPU time required increases significantly
Solution Approach 1:
The patent extracts and utilizes the bitmap structure to skip over tuples that cannot possibly contribute to the aggregate result. By performing bitwise operations on the bitmaps before accessing the actual tuple data, the system identifies and extracts only the relevant tuples that satisfy the query conditions. This extraction approach maintains the accuracy of aggregate computations while dramatically reducing the number of tuples that need to be processed, thereby lowering CPU time requirements.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Method and apparatus for performing a database query using a bitmap index in a database having a plurality of bitmap vectors, the method comprising the steps of: upon receipt of a query containing a selection condition, creating a filter bitmap based on the query input; performing a bitmap scan for the filter bitmap in order to determine at least one value of an existing matching data-base entry; determining a bitmap of the determined at least one value;creating a filtered bitmap by combining the filter bitmap with the bitmap of the determined at least one value; determining all entries in the database matching the filtered bitmap; subtracting the filtered bitmap from the filter bitmap, thus creating a new filter bitmap; repeating the steps, starting with the step of performing a bitmap scan based on the new filter bitmap, until the step of sub-tracting results in an empty bitmap.