Large-Value Transactions Using OLB Storage and MVCC Locking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in ensuring concurrent transactions in a distributed environment while maintaining data integrity and consistency, particularly in handling large values, and often suffer from issues like deadlock and starvation.
Innovation Solution
The implementation of a concurrency control mechanism combining multi-version concurrency control (MVCC) for read operations and locking for write operations, along with a two-level transaction hierarchy, ensures that transactions execute safely and efficiently, using a linearizable storage system to manage snapshot isolation and release locks upon commit.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If concurrency control mechanisms are implemented to ensure data integrity in distributed databases, then data consistency is improved, but system performance and transaction throughput deteriorate due to locking overhead and deadlock risks
Solution Approach 1:
The patent segments the database schema into multiple independent units (schema units) that can be modified independently. This segmentation allows concurrent transactions to proceed on different schema units simultaneously, reducing locking contention and improving throughput while maintaining consistency within each unit. The schema evolution operation is divided into: (1) creating a new schema unit, (2) migrating data in batches, and (3) switching the active schema unit, allowing other transactions to continue on the original schema unit during migration.
Solution Approach 2:
The patent performs preliminary actions by creating a new schema unit and copying necessary data before the original schema unit is dropped. This preliminary setup allows the system to switch schemas atomically without blocking ongoing transactions. The migration process prepares the new schema unit in advance, so when the switch occurs, transactions can continue uninterrupted on the old schema unit while the new one becomes available.
2Adaptability or versatility
If schema evolution operations are performed in distributed databases, then system adaptability is improved, but transaction conflicts and deadlocks increase due to locking requirements
Solution Approach 1:
The patent introduces a schema unit as an intermediary layer between the physical database storage and the transaction processing system. This schema unit abstraction allows schema evolution to occur without directly impacting transaction execution. The schema unit acts as a mediator that can be created, modified, and switched independently, reducing direct conflicts between schema operations and transactions. The system can have multiple schema units coexisting, with transactions routing to appropriate schema units, thereby reducing deadlock scenarios.
3Ease of manufacture
If large values are stored directly in distributed database tables, then data storage simplicity is improved, but read and write performance deteriorate due to increased I/O operations
Solution Approach 1:
The patent extracts large values from the main database table rows and stores them separately in external storage systems such as cloud storage or object storage. The database table contains only references (pointers or keys) to the external storage locations. This extraction reduces the size of database rows, improving read/write performance and reducing I/O operations. The large values are taken out from the critical path of transaction processing while remaining accessible through the stored references.
Data Source
AI summary
The subject technology receives a key-value pair. The subject technology determines a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space. The subject technology performs a write operation of a key-pointer pair into a regular space. The subject technology performs a plurality of write operations for multiple key-value parts of the key-value pair into the OLB key space. The subject technology performs conflict checking of the regular space for the key-value pair.