Disk-Based HTAP Query Processing with Partial Page Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hybrid transactional analytical processing (HTAP) databases are optimized for in-memory computation, leading to high total cost of ownership due to the high cost and power consumption of main memory, and existing cache replacement policies do not efficiently manage partial data pages.
Innovation Solution
Implementing a disk-based HTAP database using solid state devices (SSDs) like NVMe block devices, optimizing cache replacement policies for partial data pages, and employing a chunk-based approach to minimize latency and bandwidth by loading only necessary data chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If in-memory computation is used for HTAP databases, then query processing speed is improved, but total cost of ownership and power consumption increase
Solution Approach 1:
The patent segments data pages into partial data pages containing only the columns needed for query processing. This allows the system to load and process only necessary data portions in memory, reducing memory requirements and power consumption while maintaining query processing speed for analytical operations.
Solution Approach 2:
The patent implements partial data page loading where only the required columns (predicate columns and aggregate columns) are loaded into memory instead of complete data pages. This partial action approach reduces memory footprint and power consumption while still achieving the necessary query processing performance.
2Ease of operation
If complete data pages are loaded into memory, then query processing is simplified, but memory footprint and cost increase
Solution Approach 1:
The patent divides data pages into segments based on column requirements. Partial data pages are created containing only the specific columns needed for the query (predicate columns for filtering and aggregate columns for computation), eliminating the need to load unnecessary columns into memory.
Solution Approach 2:
The patent extracts only the necessary columns from complete data pages to create partial data pages. This extraction process removes unnecessary data from memory loading, reducing memory footprint while maintaining query processing capability.
3Device complexity
If conventional cache replacement policies are used, then cache management is simple, but efficiency in managing partial data pages is reduced
Solution Approach 1:
The patent implements dynamic cache replacement policies that adapt to the specific characteristics of partial data pages. The system dynamically determines which partial data pages to retain or evict based on query patterns and data access requirements, improving cache efficiency for analytical workloads while managing the complexity of partial page management.
Data Source
AI summary
A method for processing a query may include receiving a query associated with one or more predicate columns and one or more aggregate columns. To respond to the query, one or more partial data pages including the one or more predicate columns but not the one or more aggregate columns may be loaded from disk to memory. For each partial data page, a first value occupying the one or more predicate columns may be evaluated to identify one or more rows satisfying a predicate associated with the query. A portion of a data page containing the aggregate columns may be loaded from disk into memory. A result of the query corresponding to a second value occupying the aggregate columns may be generated based on the portion of the data page loaded in the memory. Related systems and articles of manufacture are also provided.