Multi-Volume Database Recovery with Targeted Transaction Replay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In modern distributed cloud services, the I/O bottleneck has shifted from individual disks and nodes to the network between the database processing layer and the storage layer, leading to performance issues and scalability challenges due to amplified traffic and dominant response times from outlier storage nodes or network paths.
Innovation Solution
Implementing a distributed database system that segregates redo log data into one set of volumes and change log data into another, utilizing efficient distributed storage, allowing for improved retrieval of committed change data without database engine intervention and reducing recovery times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If log data is stored in a single volume, then storage management is simple, but access bottlenecks occur and recovery time increases
Solution Approach 1:
The patent divides the database storage into multiple separate volumes: a primary volume for committed log data and secondary volumes for uncommitted or backup log data. This segmentation allows parallel access to different log segments, eliminating the single-volume access bottleneck and enabling faster recovery operations without requiring complex distributed storage systems.
Solution Approach 2:
The patent introduces a temporal dimension to storage organization by separating committed and uncommitted transactions into different volumes. This dimensional separation allows the system to quickly access only the necessary committed transactions during recovery, dramatically reducing recovery time while maintaining manageable storage complexity through clear volume classification.
2Productivity
If all log data is processed through the database engine, then data consistency is maintained, but access bottlenecks reduce throughput
Solution Approach 1:
The patent extracts the log retrieval function from the database engine by implementing a log reader that directly accesses the primary volume for committed log data. This extraction eliminates the database engine as a bottleneck, allowing high-throughput parallel access to log data while maintaining data consistency through the use of committed transaction markers and coordinated access protocols.
Solution Approach 2:
The patent introduces a log reader as an intermediary component between the storage volume and the database engine. This intermediary handles log data retrieval and preprocessing, offloading work from the database engine while ensuring data consistency through proper transaction commitment verification and coordinated access control mechanisms.
3Productivity
If network traffic is amplified for parallel writes, then I/O throughput increases, but network bandwidth becomes the bottleneck
Solution Approach 1:
The patent segments log write operations across multiple volumes, allowing parallel writes to different volume segments simultaneously. This segmentation increases I/O throughput by utilizing multiple storage paths while reducing network bandwidth consumption per path, as the total traffic is distributed across independent storage nodes rather than concentrated through a single network bottleneck.
Data Source
AI summary
Techniques for fast database recovery in a multi-volume database environment via transactional awareness are described. In the event of a failure associated with a first volume storing database page data, the first volume can be restored to a point in time and transactional metadata from a second volume storing logical change data can be obtained for a limited number of transactions occurring at/after that point in time, as opposed to analyzing extremely large change log files. These transactions can be checked to ensure that they have all been persisted, and if not, change data for those transactions can be obtained from the second volume and used to replay these transactions on the restored first volume.