Hierarchical Metadata for Column Database Memory Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In column-oriented databases, loading all column data into memory for query processing wastes system resources, as only a small portion of the data is typically required, and existing loading techniques lack flexibility to adapt to varying data needs over time.
Innovation Solution
A hierarchical memory management system that allows users to specify loading preferences at multiple levels (column, partition, and table) using metadata, enabling page-based or column-based loading, with a hierarchy to resolve conflicts and optimize data movement between disk storage and memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all column data is loaded into memory for query processing, then query processing can be performed, but system resources are wasted by loading unnecessary data
Solution Approach 1:
The patent segments column data into multiple partitions, allowing selective loading of only those partitions required for query processing. This divides the monolithic column data structure into manageable segments that can be independently managed and loaded based on query needs, directly resolving the contradiction between loading sufficient data for processing and avoiding waste of resources on unnecessary data.
Solution Approach 2:
The patent implements different loading strategies for different partitions within the same column based on local characteristics and query requirements. Each partition can be loaded or unloaded independently, allowing the system to optimize memory usage by maintaining only the necessary partitions in memory while keeping others on disk, thus eliminating the need to load all column data uniformly.
2Quantity of substance
If page-based loading is used to load smaller portions of data, then memory utilization improves, but data loading and unloading operations become more frequent
Solution Approach 1:
The patent implements dynamic partition loading where partitions are loaded into and unloaded from memory based on real-time query requirements and system state. The system can adaptively adjust which partitions reside in memory versus disk storage, optimizing the balance between memory utilization and loading efficiency by maintaining frequently accessed partitions in memory while using efficient page-based loading for less frequently accessed data.
3Speed
If column-based loading is used to load data quickly as a single unit, then data loading speed improves, but additional memory is required for data that may not be needed
Solution Approach 1:
The patent segments column data into partitions that can be loaded using efficient column-based loading when entire columns are needed, while allowing selective loading of individual partitions when only portions are required. This segmentation enables the system to leverage fast column-based loading for bulk operations while avoiding the memory overhead of loading complete columns when partition-level selectivity is sufficient.
Data Source
AI summary
System, method, and various embodiments for providing a metadata based memory management system are described herein. An embodiment operates by determining metadata of a table of a database indicating whether data of the table is to be stored in memory or disk storage, wherein the table comprises a plurality of columns and a plurality of rows. A first column of the table is identified. A hierarchy of the metadata for the table including a column value, partition value, and table value is determined, wherein the column value is prioritized over the partition value, and wherein the partition value is prioritized over the table value. The data of the table stored in the column is loaded from the disk storage into the memory based on the hierarchy. A query is processed based on the data loaded into the memory.