Database Transaction Recovery via Snapshot-Based Cache Batching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database management systems face inefficiencies when handling large objects in remote object stores due to high input/output latency, which can be exacerbated by writing small data pieces directly to these stores, leading to suboptimal performance and increased latency.
Innovation Solution
Implementing a snapshot-based mechanism where write data is collected in a local cache and then written in larger chunks to the remote object store, utilizing a snapshot control engine to manage this process, thereby reducing latency and improving performance by minimizing direct writes to the object store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If small data pieces are written directly to the remote object store, then data is stored immediately, but I/O latency increases and performance deteriorates
Solution Approach 1:
The patent implements a buffer that temporarily holds write data before it is flushed to the remote object store. This preliminary buffering action allows multiple small writes to be accumulated and then flushed together as larger batches, reducing the frequency of I/O operations to the remote store and thereby reducing I/O latency while maintaining data integrity through the buffering mechanism
Solution Approach 2:
The patent introduces a buffer as an intermediary component between the database server and the remote object store. This buffer mediates the data flow by receiving write operations from the database server and managing the timing and batching of writes to the remote store, thus decoupling the immediate write requirement from the actual I/O operation and reducing latency
2Productivity
If write data is buffered in local cache before writing to remote object store, then I/O latency is reduced, but system complexity increases
Solution Approach 1:
The patent implements a snapshot-based recovery mechanism where periodic snapshots of the buffer contents are stored. When a failure occurs, the system can recover by restoring from the last successful snapshot and replaying the buffered writes, thus managing the complexity of buffer persistence through a recoverable snapshot approach rather than requiring complex distributed transaction protocols
Solution Approach 2:
The patent creates snapshots (copies) of the buffer contents at periodic intervals or upon certain events. These snapshots serve as recovery points and allow the system to manage buffer state without requiring complex real-time synchronization, simplifying the buffer management complexity while maintaining productivity through efficient recovery capabilities
3Reliability
If snapshots are taken frequently to ensure data recovery, then data integrity is improved, but storage overhead increases
Solution Approach 1:
The patent implements snapshots based on triggers such as buffer fill thresholds or time intervals rather than after every write operation. This partial action approach takes snapshots only when necessary to maintain recovery capability, avoiding excessive storage overhead while still providing adequate recovery points for data integrity
Solution Approach 2:
The patent allows the snapshot frequency and retention policy to be configured as adjustable parameters. By changing these parameters based on workload characteristics and recovery requirements, the system can optimize the balance between recovery capability and storage overhead, taking snapshots more frequently when reliability is critical and less frequently when storage is constrained
Data Source
AI summary
In some examples, a system sends a transaction to a database server to cause storing of data of the transaction in a cache of the database server, where the data in the cache is for inclusion in a backup of data from the database server to a remote data store (e.g., the backup may be in a cloud and may be a snapshot). The system detects a failure associated with the database server, and in response to detecting the failure, requests, from the database server or a replacement database server, transaction information of at least one transaction that was successfully applied to the remote data store, the transaction information based on the backup of data. The system causes replay one or more transactions to recover data at the database server or the replacement database server, to perform recovery of the database server or the replacement database server to a current state.