Columnar Cache for HTAP Reads Without OLTP Degradation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based data warehouses face challenges in managing transactional processing that can rapidly grow, making it difficult to compact data securely without affecting accuracy or integrity, especially when large analytical reads are performed, which can degrade OLTP database performance and cause offline issues.
Innovation Solution
A hybrid system employing a blob manager and blob workers converts key-value data into snapshot and delta files, allowing efficient handling of large analytical-style reads by replicating data into blob storage and using pointers for reconstruction, thereby separating read requests from OLTP databases to maintain performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large analytical reads are performed on OLTP databases, then analytical query capabilities are improved, but database performance degrades and offline issues occur
Solution Approach 1:
The patent segments the database system into two distinct components: an OLTP database for transactional operations and a data warehouse for analytical queries. This segmentation allows analytical reads to be directed to the data warehouse, preventing performance degradation of the OLTP database while maintaining both transactional and analytical capabilities.
Solution Approach 2:
The patent introduces a data warehouse as an intermediary layer between the OLTP database and analytical query requirements. This intermediary handles all analytical read operations, shielding the OLTP database from the performance impact of large analytical queries while still enabling comprehensive analytical capabilities.
2Quantity of substance
If data is compacted to manage growth, then storage efficiency is improved, but data accuracy or integrity may be affected
Solution Approach 1:
The patent extracts the data management burden from the OLTP database by replicating data into a separate data warehouse. This extraction allows the OLTP database to focus on maintaining transactional data integrity while the data warehouse handles large-scale data storage and compacting operations.
3Speed
If data is replicated into blob storage, then read performance is improved, but system complexity increases
Solution Approach 1:
The patent adds a new dimension to the storage architecture by introducing blob storage as a separate layer for data replication. This dimensional addition enables high-performance analytical reads from blob storage while the OLTP database maintains its transactional integrity, distributing the workload across multiple storage dimensions.
Data Source
AI summary
The subject technology receives, by an execution node, blob metadata from a key-value store, the blob metadata including information related to a set of blob files. The subject technology determines, by the execution node using the blob metadata, whether a copy of each of the set of blob files is stored in a local cache of the execution node. The subject technology transforms at least one blob file, retrieved from a blob store, to a second file in a column file format, the at least one blob file being in a first format that is different than the column file format, the transforming comprising at least converting a particular snapshot file from the at least one blob file to a particular set of rowsets and writing the set of rowsets into the second file in the column file format. The subject technology stores the second file in the local cache.