Database Scan Optimization via Bitmap Index Columns
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
SCAN operations in databases become increasingly resource-intensive and time-consuming as payload sizes increase, due to the need to read each value from disk to RAM, especially in column-family based key value stores.
Innovation Solution
An index column is introduced as an additional column in the database table, storing bitmaps that indicate the presence or absence of data in each column, allowing SCAN operations to be performed efficiently by accessing the index column instead of traditional scanning, thereby reducing memory usage and computing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional SCAN operation reads each value from disk to RAM, then complete data retrieval is achieved, but memory consumption and computing time increase significantly with payload size
Solution Approach 1:
The patent segments the data retrieval process into two parts: first retrieve only the bitmap index from disk, then use the bitmap to determine which actual data values need to be accessed. This segmentation allows the SCAN operation to avoid loading entire payloads into RAM, significantly reducing memory consumption while maintaining accurate scan results.
Solution Approach 2:
The patent extracts the presence/absence information from the full data values and stores it separately in the bitmap index. This extraction allows the system to work with a minimal representation (the bitmap) during SCAN operations rather than handling complete payloads, thereby reducing memory usage and computing time.
2Measurement precision
If traditional SCAN operation reads each value from disk to RAM, then complete data retrieval is achieved, but computing time increases with payload size
Solution Approach 1:
The patent segments the data retrieval process into two parts: first retrieve only the bitmap index from disk, then use the bitmap to determine which actual data values need to be accessed. This segmentation allows the SCAN operation to avoid loading entire payloads into RAM, significantly reducing memory consumption while maintaining accurate scan results.
Solution Approach 2:
The patent performs preliminary action by pre-computing and storing bitmap indexes that indicate the presence or absence of values in each column. This preliminary preparation allows SCAN operations to quickly determine which data needs to be retrieved without examining actual payload sizes, thereby reducing computing time.
3Productivity
If bitmap index column is added to database table, then SCAN operation efficiency is improved, but device complexity increases
Solution Approach 1:
The patent implements a universal bitmap index structure that can serve multiple purposes: it optimizes SCAN operations, supports GET operations, and works with any column in the table. This multi-functionality justifies the added complexity by providing broad benefits across different database operations rather than solving a single narrow problem.
Data Source
AI summary
SCAN operations for databases where scan time is dependent on a payload size consume too much memory space and computing time as payload sizes increase. A database table is configured to include an additional index mapping column that stores bitmaps related to the corresponding row of the table. Each bit in the bitmap corresponds to a column and indicates whether that column stores a value. Inclusion of an index column in a table decouples the time it takes to perform the SCAN operation on a column from the payload size of data stored in the column. The bitmaps stored in the index column are relatively small and uniform in size, so the SCAN operation on such a database requires only for the bitmap values of the applicable rows to be obtained from the index column and inspected.