Disk-Based HTAP Database Using PAX Format and Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional databases are optimized for either transactional processing or analytical processing, leading to high total cost of ownership due to reliance on in-memory computation, which is costly and power-intensive, and lacks efficient support for both transactional and analytical workloads.
Innovation Solution
Implementing a hybrid transactional analytical processing (HTAP) database on disk using solid state devices (SSDs) like NVMe, which provides optimal throughput and capacity while reducing memory usage, enabling efficient transactional and analytical processing by storing data in a partition attribute across (PAX) format with metadata for direct access and caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If in-memory computation is used to achieve fast processing, then processing speed is improved, but cost and power consumption increase
Solution Approach 1:
The patent segments data storage and processing into hot data (frequently accessed) and cold data (infrequently accessed). Hot data is kept in memory for fast access, while cold data is stored on disk. This segmentation allows the system to achieve fast processing for critical operations while reducing memory usage and power consumption by moving less critical data to disk storage.
Solution Approach 2:
The patent changes the storage medium parameter from exclusively in-memory to a hybrid memory-disk architecture. By adjusting which data resides in memory versus disk based on access patterns and importance, the system optimizes the balance between processing speed and power consumption, achieving fast access when needed while conserving energy during normal operations.
2Speed
If in-memory computation is used to achieve fast processing, then processing speed is improved, but total cost of ownership increases
Solution Approach 1:
The patent segments the data storage architecture into memory and disk components, allowing the system to achieve fast processing speeds for critical data while using cost-effective disk storage for the majority of data. This segmentation reduces the total amount of expensive memory required, thereby lowering the total cost of ownership while maintaining acceptable processing performance.
Solution Approach 2:
The patent employs disk storage, which is significantly cheaper than memory, for storing cold and archival data. While disk access is slower, it provides cost-effective storage capacity that reduces the overall system cost. The hybrid architecture allows the system to use expensive memory only when absolutely necessary for fast processing, while relying on inexpensive disk storage for the bulk of data retention.
3Productivity
If conventional databases are optimized for either transactional or analytical processing, then specialization is improved, but versatility deteriorates
Solution Approach 1:
The patent implements a universal database architecture that can handle both transactional (OLTP) and analytical (OLAP) workloads within the same system. The hybrid memory-disk storage architecture provides the flexibility to support different query patterns and workload types, allowing a single database system to serve multiple functions rather than requiring separate specialized systems.
Solution Approach 2:
The patent employs dynamic data placement and caching strategies that adapt to different workload types. The system can dynamically adjust which data resides in memory versus disk based on access patterns, allowing it to optimize performance for transactional workloads when needed and for analytical workloads at other times, providing versatile support for different processing requirements.
4Speed
If entire data pages are loaded into memory for query execution, then query access speed is improved, but memory usage increases
Solution Approach 1:
The patent extracts only the necessary portions of data pages into memory based on query requirements and data importance, rather than loading entire data pages. This selective extraction allows the system to maintain fast query access speeds for critical data while minimizing memory usage by leaving less critical data on disk.
Solution Approach 2:
The patent applies partial loading of data pages into memory, loading only the portions needed for current query operations. This partial action approach achieves sufficient query access speed without the excessive memory consumption that would result from loading complete data pages, optimizing the balance between performance and resource utilization.
Data Source
AI summary
A method for providing optimized support for transactional processing and analytical processing with minimal memory footprint may include storing, on a data page in a disk of a database system, a portion of one or more columns of data from a database table. A metadata associated with the data page may be stored on a metadata page in the disk of the database system. The metadata may include one or more byte ranges on the data page at which the portion of the one or more columns of data is stored. The database system may execute one or more queries by accessing, based at least on the metadata associated with the data page, a portion of the data page storing the portion of the one or more columns of data required by the one or more queries. Related systems and articles of manufacture are also provided.