Transaction Aggregator Layer for Database Concurrency Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database scaling techniques often lead to input/output bottlenecks, causing database unavailability due to high demand, especially with large transactions that exceed the capacity of the host system, resulting in reduced scalability and performance.
Innovation Solution
Implementing a transaction aggregator layer between clients and databases to enqueue data manipulation requests and commit them in a single transaction, reducing the load on the database by aggregating requests and optimizing input/output operations, allowing for better resource management and load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional horizontal or vertical scaling techniques are used to increase database capacity, then the database can store more data, but the input/output capacity is exceeded and the database becomes unavailable
Solution Approach 1:
The patent segments the transaction processing into two distinct phases: a read phase where multiple clients can read data concurrently without locking, and a write phase where a single writer performs the actual data modification. This segmentation allows the database to handle multiple concurrent operations without exceeding I/O capacity, as the read operations do not block the write operation and vice versa.
Solution Approach 2:
The patent introduces an intermediary mechanism (the shared data structure with versioning) between multiple readers and the single writer. This intermediary allows readers to access data without directly competing with the writer for I/O resources, as readers work with cached copies and the writer operates on the underlying data structure with proper synchronization.
2Quantity of substance
If long running transactions are used to process large amounts of data, then significant data manipulation can be performed, but clients are locked from the database for large amounts of time
Solution Approach 1:
The patent segments the transaction into a preparation phase (where the writer reserves the write lock and prepares data changes in memory) and an execution phase (where changes are applied to the shared data structure). During the preparation phase, multiple readers can still access the database, and the actual lock duration is minimized to only the execution phase, significantly reducing client lock time while still handling large data volumes.
Solution Approach 2:
The writer performs preliminary actions (data validation, transformation, and reservation of locks) before actually modifying the shared data structure. This preliminary preparation allows the actual write operation to complete quickly, minimizing the time clients are locked out while ensuring data integrity and consistency.
3Productivity
If multiple clients access the database concurrently, then database throughput increases, but input/output bottlenecks prevent scaling
Solution Approach 1:
The patent segments database operations into read-heavy and write-heavy paths. Readers can access cached data in memory without generating I/O overhead, while writers perform batched updates to the shared data structure. This segmentation allows high throughput for read operations with minimal I/O, and efficient batch processing for writes, overall reducing I/O overhead while maintaining high throughput.
Solution Approach 2:
The patent maintains continuous useful action by allowing readers to continuously access data from the shared structure without blocking, while the writer continuously prepares and applies updates. The read-write lock mechanism ensures that readers can operate continuously on cached data while the writer operates continuously on the underlying structure, maximizing throughput without I/O bottlenecks.
Data Source
AI summary
To achieve long running batch updates, a transaction aggregator layer and one or more data aggregation queues, such as a command queue, are placed between one or more clients and a database. Writes go to the database aggregator layer and are enqueued in the appropriate data aggregation queue, which may be allocated by transaction. Committing the transaction drains the queue and writes the contents of the queue in one transaction-like update to the database. By using the transaction logic in the transaction aggregator layer, transaction-like behavior may be achieved that allows both updates and transactions to scale with less impact on database servers.