Hybrid Database Serialization for Concurrent Transaction Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based data warehouses and database systems face challenges in efficiently processing concurrent transactions due to rapid data growth and the need for scalable and performant transactional processing, especially in handling different data formats and ensuring data integrity and consistency in distributed environments.
Innovation Solution
A hybrid database architecture that employs a combination of Online Transactional Processing (OLTP) and Online Analytical Processing (OLAP) systems, utilizing a serializer system to convert columnar data to row data for OLTP systems, and implementing a two-level transaction hierarchy with multi-version concurrency control and locking mechanisms for write operations, along with bitmap compression and auxiliary compression techniques to optimize data storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If transactional processing is implemented in cloud-based data warehouses to enable additional operations, then system functionality and versatility are improved, but data processing complexity and difficulty increase
Solution Approach 1:
The patent segments the database architecture into distinct OLTP and OLAP layers, with the OLTP layer handling transactional operations and the OLAP layer handling analytical queries. This segmentation allows the system to support multiple operation types without increasing overall complexity, as each layer is optimized for its specific function.
Solution Approach 2:
The patent introduces a data serialization layer as an intermediary between OLTP and OLAP systems. This intermediary converts data between row-based format (for OLTP) and columnar format (for OLAP), enabling seamless interaction between the two layers without directly coupling them, thus managing complexity while maintaining versatility.
2Productivity
If concurrent transactions are processed rapidly to improve productivity, then output per unit time increases, but data consistency and integrity become harder to maintain
Solution Approach 1:
The patent implements preliminary actions through transaction logging and version recording before actual data modifications. Each transaction is logged with its version information and dependencies established in advance, allowing the system to maintain consistency through pre-planned conflict resolution strategies rather than reactive measures.
Solution Approach 2:
The patent employs feedback mechanisms through transaction version tracking and conflict detection. When concurrent transactions access the same data, the system detects conflicts and provides feedback to resolve them through serialization or version control, ensuring data integrity while allowing high concurrency throughput.
3Adaptability or versatility
If data is stored in different formats to support diverse operations, then system adaptability improves, but data retrieval and processing difficulty increase
Solution Approach 1:
The patent changes the organizational parameter of data storage by maintaining row-based format in OLTP systems for efficient transactional updates and columnar format in OLAP systems for efficient analytical queries. The data serialization layer automatically transforms between these parameter configurations, allowing the system to optimize for different operational modes without manual intervention.
4Speed
If distributed database architecture is used to enable parallel processing, then processing speed improves, but system complexity and coordination overhead increase
Solution Approach 1:
The patent segments the distributed database into autonomous OLTP nodes and OLAP nodes that can operate independently in parallel. Each node manages its own data and transactions locally, minimizing coordination overhead while maintaining overall system consistency through the serialization layer that handles inter-node data transformations.
Data Source
AI summary
The subject technology serializes, by at least one hardware processor, non-primary key data of column-organized data into compressed serialized value data that is in a row-organized sequence, the compressed serialized value data compressed using at least one bitmap, the non-primary key data comprising a schema identifier, the column-organized data being stored in a columnar database system, the column-organized data comprising primary key data and the non-primary key data. The subject technology stores the compressed serialized value data in a key-value data store of a key-value database system, the key-value database system processing key-value data in a key-value format. The subject technology receives a query by the columnar database system. The subject technology deserializes a portion of the compressed serialized value data that corresponds to the query. The subject technology processes the query using the columnar database system.