Database Load Unit Conversion Using Unified Persistence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in handling large data stores due to memory limitations, as traditional methods for switching between column-loadable and page-loadable formats require costly memory and processing resources through full rewrite operations.
Innovation Solution
Implementing a unified persistence format that allows load unit conversion without rewriting data in persistence storage, using separate composite page chains for data, dictionary, and index subcomponents, enabling dynamic format switching based on workload and operating conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional methods are used for switching between column-loadable and page-loadable formats, then load unit conversion is achieved, but costly memory and processing resources are consumed through full rewrite operations
Solution Approach 1:
The patent segments the database object into separate subcomponents (data subcomponent, dictionary subcomponent, index subcomponent) that can be independently managed. Each subcomponent has its own page chain that can be selectively loaded or shared, avoiding the need to rewrite entire objects during load unit conversions.
Solution Approach 2:
The patent implements dynamic load unit conversion by allowing the database system to switch between column-loadable and page-loadable formats based on current workload and operating conditions. The unified persistence format enables flexible switching without requiring physical rewrites, as the same persistence structure supports both load units through dynamic page chain sharing strategies.
2Adaptability or versatility
If full rewrite operations are performed for load unit conversion, then format switching is achieved, but processing time and resource consumption increase
Solution Approach 1:
The patent prepares the persistence store in advance by organizing data into a unified persistence format with separate page chains for each subcomponent. This preliminary structuring enables rapid load unit conversion without time-consuming rewrites, as the necessary page chains are already in place and can be selectively shared or loaded based on current needs.
Solution Approach 2:
The patent uses page chain sharing mechanisms that allow virtual copying or referencing of page chains without physical duplication. When converting between load units, the system can share existing page chains across different load unit representations, avoiding the need to copy or rewrite entire data structures and significantly reducing processing time.
3Ease of operation
If separate page chains are used for each subcomponent, then page-loadable format is achieved, but memory usage increases compared to column-loadable format
Solution Approach 1:
The patent merges the persistence structures by using a unified persistence format where separate page chains for data, dictionary, and index subcomponents are managed together. This unified structure enables efficient memory usage by allowing page chain sharing across different load units, combining the benefits of separate subcomponent management with the memory efficiency of integrated storage.
Data Source
AI summary
A database system detects a query targeting a first database object stored in a unified persistence format in a persistence store, where a first load unit is specified for loading the first database object into an in-memory store. The database system determines whether to convert the first load unit to a second load unit based on one or more conditions. In response to determining to convert the first load unit to the second load unit for the first database object, the first database object is loaded into the in-memory store according to the second load unit without performing a rewrite of the first database object in the persistence store. In an example, the first load unit is a column-loadable unit and the second load unit is a page-loadable unit.