Hybrid MVCC Data Storage Format Transition for Query Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage and retrieval systems for big data face inefficiencies in handling both transactional and analytical queries, leading to data duplication, stale data, high maintenance costs, and resource wastage, as they require separate database management systems and complex ETL processes.
Innovation Solution
A hybrid data storage and retrieval system that uses Multi-Version Concurrency Control (MVCC) row format, transitioning data to MVCC columnar format and then columnar format, allowing non-overlapping data storage in multiple formats for efficient execution of both transactional and analytical queries, with query routing to appropriate node clusters based on query type.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate database management systems are used for transactional and analytical queries, then query handling capability is improved, but system complexity and maintenance costs increase
Solution Approach 1:
The patent merges transactional and analytical query handling into a single database management system by implementing multiple storage formats (row-oriented for transactions, columnar for analytics) within the same system architecture, eliminating the need for separate systems while maintaining both capabilities
Solution Approach 2:
The database system is designed to be universal by supporting multiple query types (transactional and analytical) through a unified architecture that can dynamically select appropriate storage formats and query execution paths based on the query type, making the system adaptable to different workloads
2Productivity
If data is stored in multiple formats for different query types, then query performance is improved, but data duplication and storage overhead increase
Solution Approach 1:
The patent segments data storage into different formats (row-oriented and columnar) based on query type requirements, where transactional queries access row-oriented stored data and analytical queries access columnar stored data, allowing efficient querying without requiring all data to be duplicated in all formats simultaneously
Solution Approach 2:
The system changes the storage format parameter of data based on the query type, transforming or selecting between row-oriented and columnar formats dynamically, which allows the same data to serve different query purposes without permanent duplication in both formats
3Adaptability or versatility
If complex ETL processes are implemented for data transformation, then data processing capability is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent performs preliminary data transformation by maintaining data in multiple storage formats (row-oriented and columnar) in advance, so when queries arrive, the appropriate format is already prepared and readily accessible, eliminating the need for time-consuming ETL processes at query time
Data Source
AI summary
A method includes receiving data in a first format to be written into a data storage, where the first format is a multiversion concurrency control (MVCC) row format. The data is stored in the first format in the data storage and is transitioned from the first format to a second format based on a first transition parameter, where the second format is an MVCC columnar format. Data is stored in the second format in the data storage and is transitioned from the second format to a third format based on a second transition parameter, where the third format is a columnar format. The data storage stores non-overlapping data in the first format, the second format, and the third format.