Unified Table Delta Dictionary Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems face challenges in optimizing memory usage and load times for dictionary blocks storing data from multiple columns, particularly in handling variable-sized data, which can lead to inefficient data copying and increased storage overhead.
Innovation Solution
The method involves loading page data from persistent storage into a page buffer, copying dictionary values into in-memory arrays, and deallocating the page data, while pinning variable-sized data pages to maintain efficient memory usage and reduce storage overhead by using re-direction references for variable-sized data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If dictionary blocks for multiple columns are stored together in unified table delta, then storage overhead is reduced and memory efficiency is improved, but data copying complexity increases and load time optimization becomes more difficult
Solution Approach 1:
The patent segments dictionary blocks by column type, separating fixed-size dictionary blocks from variable-size dictionary blocks. This segmentation allows efficient copying of fixed-size blocks while handling variable-size blocks through re-direction references, resolving the contradiction between unified storage benefits and copying complexity
Solution Approach 2:
The patent introduces re-direction references as intermediaries for variable-size data, where instead of copying actual variable-size dictionary values, the system copies references that point to the actual data locations. This intermediary approach reduces copying complexity while maintaining the benefits of unified storage
2Loss of time
If variable-sized data is stored using re-direction references, then memory usage is reduced and load time is decreased, but implementation complexity increases
Solution Approach 1:
The patent extracts variable-size data from the main dictionary block structure and stores it separately, using re-direction references to point to these extracted data locations. This extraction allows the main dictionary block to contain only fixed-size data and references, significantly reducing load time while the separation of variable-size data manages implementation complexity
Solution Approach 2:
The patent performs preliminary organization of variable-size data into separate structures before integrating them with the main dictionary through re-direction references. This preliminary action simplifies the overall implementation by pre-establishing the reference mechanism rather than handling variable-size data integration during normal operations
3Reliability
If page data is loaded into page buffer and then copied to in-memory array, then memory management is optimized and data is properly allocated, but additional copying operations increase processing time
Solution Approach 1:
The patent uses selective copying, copying only fixed-size dictionary block contents to in-memory arrays while leaving variable-size data references in place. This selective copying approach maintains proper memory management and data allocation while minimizing the copying operations required, thus reducing processing time compared to copying all data
Data Source
AI summary
First page data from one or more pages stored on a persistent storage can be loaded into a page buffer in a main system memory of one or more computing systems, and second page data that includes first dictionary values of a first dictionary for a first database column can be loaded and pinned into the page buffer in the main system memory. First contents of a first dictionary block containing first re-direction references to the first dictionary can be copied from the loaded first page data into a first in-memory array, and the first page data can be deallocated from the page buffer. The first re-direction references can direct to the first dictionary values of the first dictionary in the pinned second page data.