HTAP Schema Evolution with Columnar Cache for Analytical Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based database systems face challenges in efficiently handling large analytical read requests without degrading transactional processing performance, particularly in hybrid transactional/analytical processing (HTAP) workloads, due to the complexity of OLAP queries impacting OLTP databases.
Innovation Solution
A hybrid system that employs a blob manager and blob workers to convert key-value data into blob data, utilizing snapshot and delta files to manage large analytical-style read requests by sending pointer data to clients for reconstruction, thereby optimizing performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the database system processes large analytical read requests directly on the OLTP database, then analytical query capabilities are provided, but transactional processing performance degrades and the database may go offline
Solution Approach 1:
The patent segments the database system into separate OLTP and OLAP components. The OLTP database handles transactional workloads while the OLAP database handles analytical workloads. This segmentation allows each component to optimize for its specific workload type without interfering with the other, resolving the contradiction between providing analytical capabilities and maintaining transactional performance.
Solution Approach 2:
The patent introduces an intermediary layer that includes a data synchronization mechanism and query routing component. This intermediary transfers data between OLTP and OLAP databases and routes queries appropriately, allowing analytical requests to be handled by the OLAP database without directly impacting the OLTP database performance.
2Reliability
If the database system compacts data in a secure manner to maintain integrity, then data security is improved, but the compaction process affects accuracy or integrity of the data
Solution Approach 1:
The patent performs preliminary validation and verification actions before data compaction. Checksum verification, data lineage tracking, and pre-compaction backups are executed to ensure data integrity is maintained throughout the compaction process, preventing accuracy loss while achieving security improvements.
3Loss of information
If the database system handles large analytical reads by reading all data, then complete data retrieval is achieved, but response time increases and bottlenecks occur
Solution Approach 1:
The patent implements partial action by using selective data retrieval mechanisms. The system retrieves only the specific portions of data needed for analytical queries from the OLAP database using targeted scan operations, rather than reading all data. This approach maintains data completeness for the required analysis while significantly reducing response time and avoiding bottlenecks.
Data Source
AI summary
The subject technology receives a request to perform a table scan operation of a table. The subject technology determines that the table is being accessed for an initial time. The subject technology populates a columnar cache with data of the table provided by the table scan operation. The subject technology determines a set of schema versions of a set of rows from the data of the table. The subject technology determines schema information of each schema from the set of schema versions. The subject technology generates a result rowset and a second rowset comprising a union of columns that have appeared at least once in each row. The subject technology performs deserialization of rows from the result rowset and the second rowset. The subject technology provides the rows from the result rowset and the second rowset to write to a file in a particular format.