Column-Level Invalidation Bitmap for In-Memory Database Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in querying data due to row-level invalidation, where it is not possible to determine which in-memory data items are not stale, leading to unnecessary retrieval from disk instead of volatile memory.
Innovation Solution
Implementing column-level invalidation information in addition to row-level information, using a column-bitmap to identify valid data items within an invalid row, allowing the database server to determine if an in-memory data item is not stale and enabling its use without reverting to disk retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If row-level invalidation is used to track stale data, then data correctness is ensured, but query efficiency deteriorates because entire rows must be reloaded from disk even when only some columns are stale
Solution Approach 1:
The patent segments the invalidation tracking from row-level to column-level granularity. Instead of marking entire rows as invalid, the system maintains separate invalidation bitmaps for each column, allowing individual columns to be marked as stale independently. This enables the database to retrieve only the valid columns from the in-memory buffer rather than reloading entire rows from disk, thus resolving the contradiction between ensuring data correctness and maintaining query efficiency.
2Speed
If in-memory data is used to speed up queries, then access speed improves, but data staleness cannot be precisely determined with row-level tracking
Solution Approach 1:
The patent divides the invalidation information into column-specific segments through separate invalidation bitmaps. Each bitmap tracks the staleness status of its corresponding column independently, providing precise information about which columns are stale. This allows the system to confidently use in-memory data for non-stale columns while identifying exactly which columns need refresh, thereby maintaining both access speed and stale data identification accuracy.
Solution Approach 2:
The system implements feedback mechanisms where column invalidation bitmaps continuously provide information about data staleness status. When columns are updated, the corresponding bitmap entries are set to indicate invalidation. This feedback enables the database server to make informed decisions about whether to use in-memory data or retrieve from disk, ensuring both speed and accuracy in staleness determination.
Data Source
AI summary
Techniques are provided for maintaining and using row-level and column-level invalidation information for a copy of data items stored in-memory via a row-bitmap and column-bitmap. The row-bitmap and column-bitmap can be used to determine whether a database query can be processed using in-memory data or not. The row-bitmap and column-bitmap can thus improve the frequency with which database queries can be processed using in-memory data.