Concurrent Transaction Log Replay for Dataset Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for recovering a dataset after a system failure involve replaying all not-yet-completed transactions before restoring client access, leading to delays that can result in timeouts and errors due to increased recovery time.
Innovation Solution
A method that parses records in the transaction log to create a dependency graph and performs on-demand recovery of specific blocks, allowing client access while concurrently replaying not-yet-completed transactions in the background, using a first-in first-out queue and multiple threads for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all not-yet-completed transactions are replayed before restoring client access, then dataset consistency is ensured, but client access time is delayed
Solution Approach 1:
The patent segments the transaction log into multiple segments and processes them concurrently using multiple threads. Each thread handles a specific segment of the log, allowing parallel replay operations. This segmentation enables the system to maintain dataset consistency while reducing the total replay time, thus resolving the contradiction between reliability and access time.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and organizing transaction logs into segments before the actual replay process. This preparation work is done in advance to enable faster concurrent processing during recovery, thereby reducing the time clients need to wait for access while ensuring consistency through structured replay operations.
2Productivity
If multiple threads are used for concurrent log replay, then processing speed is improved, but system complexity increases
Solution Approach 1:
The patent divides the transaction log into multiple manageable segments that can be processed by different threads simultaneously. This segmentation strategy increases processing speed by enabling parallel operations while keeping each thread's workload manageable, thus improving productivity without excessively increasing system complexity.
Solution Approach 2:
The patent introduces intermediary data structures such as segment headers, footers, and metadata that facilitate coordinated processing among multiple threads. These intermediaries manage the complexity of concurrent operations by providing standardized interfaces and synchronization mechanisms, allowing high-speed parallel processing while maintaining system organization and controllability.
3Productivity
If the entire transaction log is read into memory, then replay processing is accelerated, but memory usage increases
Solution Approach 1:
The patent segments the transaction log and loads only the necessary segments into memory during concurrent replay operations. This approach accelerates processing by keeping active segments in fast memory while storing less frequently accessed segments on disk, thereby improving productivity while controlling memory consumption through selective loading.
Solution Approach 2:
The patent implements partial loading of the transaction log into memory, loading only the segments that are currently needed for replay processing. This partial action approach provides sufficient processing speed for active segments while avoiding the excessive memory usage that would result from loading the entire log, thus resolving the contradiction between productivity and resource consumption.
Data Source
AI summary
A dataset is recovered after a server reboot while clients access the dataset. In response to the reboot, not-yet-completed transactions in a log are parsed to create, for each of the dataset blocks modified by these active transactions, a respective block replay list of the active transactions that modify the block. Once the block replay lists have been created, clients may access specified blocks of the dataset after on-demand recovery of the specified blocks. The on-demand recovery is concurrent with a background recovery task that replays the replay lists. To accelerate log space recovery, the parsing of the log inserts each replay list into a first-in first-out queue serviced by multiple replay threads. The queue can also be used as the cache writeback queue, so that the cache index is used for lookup of the replay list and the recovery state of a given block.