Storage Record Engine Interleaved Checkpoint Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems face performance degradation and unresponsiveness due to high bandwidth requirements for committing transactions to disk, which hinders concurrent processing of write transactions and effective backup and recovery operations, leading to potential data loss and prolonged system recovery.
Innovation Solution
Implementing a storage record engine that interleaves transaction log replay with checkpoint metadata to avoid preemption of normal storage device activity during recovery, allowing for concurrent processing of write transactions and efficient recovery without degrading storage performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transactions are committed to disk with high bandwidth, then data reliability is improved, but storage performance and system responsiveness deteriorate
Solution Approach 1:
The patent segments the commit process into periodic batches rather than continuous high-bandwidth commits. Transaction logs are accumulated and committed in periodic intervals, allowing normal storage operations to proceed concurrently without being preempted by full-bandwidth commit operations. This segmentation resolves the contradiction by maintaining reliability through periodic commits while preserving storage performance during normal operations.
Solution Approach 2:
The patent implements preliminary action by maintaining transaction logs in memory before committing to disk. Write transactions are first recorded in transaction logs with sequence identifiers, allowing them to be buffered and committed periodically rather than immediately. This preliminary logging action ensures data reliability is captured while avoiding immediate disk I/O that would degrade storage performance.
2Reliability
If transaction logs are replayed during recovery, then data integrity is restored, but normal storage device activity is preempted
Solution Approach 1:
The patent applies preliminary action by interleaving checkpoint metadata into the transaction log before recovery is needed. During normal operations, checkpoints are written that mark safe recovery points. During recovery, the system can quickly identify the latest checkpoint and replay only subsequent transactions, rather than replaying the entire log. This preliminary checkpointing action ensures data integrity while minimizing recovery time and reducing preemption of storage device activity.
Solution Approach 2:
The patent segments the recovery process by using checkpoint metadata to divide the transaction log into manageable sections. Instead of replaying all transactions from the beginning of the log, the system identifies the latest checkpoint and replays only the segment of transactions that occurred after that checkpoint. This segmentation of the recovery process maintains data integrity while significantly reducing the time and resources required, thereby minimizing preemption of normal storage device activity.
3Productivity
If checkpoint metadata is interleaved with transaction records, then recovery efficiency is improved, but record organization complexity increases
Solution Approach 1:
The patent merges checkpoint metadata directly into the transaction log stream, interleaving checkpoint records with transaction records in the same data structure. This combining approach allows the system to maintain a unified record organization rather than separate structures for transactions and checkpoints. The merged structure improves recovery efficiency by allowing the system to scan a single log and automatically identify checkpoint boundaries, while the complexity is managed through consistent record formatting and sequence identifiers.
Solution Approach 2:
The patent uses sequence identifiers as intermediaries to organize the interleaved records. Each transaction record and checkpoint record is tagged with a sequence identifier that maintains their relative ordering. This intermediary mechanism allows the system to efficiently parse and process interleaved records during recovery by following the sequence identifiers, resolving the organization complexity through a simple ordering rule rather than complex data structures.
Data Source
AI summary
A storage record engine implemented on a storage system is provided. The storage record engine further organizes hosted storage of the storage system into superblocks and chunks organized by respective metadata, the chunks being further organized into chunk segments amongst superblocks. Persistent storage operations may cause modifications to the metadata, which may be recorded in a transaction log, records of which may be replayed to commit the modifications to hosted storage. The replay functionality may establish recovery of data following a system failure, wherein replay of records of transaction logs in a fashion interleaved with checkpoint metadata avoids preemption of normal storage device activity during a recovery process, and improves responsiveness of the storage system from the perspective of end devices.