Micro-batch Snapshot Deletion Reducing Transaction Journal Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing techniques for deleting snapshots in storage area networks (SANs) face performance issues due to increased transaction journaling, which consumes memory resources and can lead to system delays and reduced availability, especially when dealing with a large number of snapshots.
Innovation Solution
The implementation of micro-batch processing for snapshot deletion, where data blocks exclusively owned by snapshots are collected and split into smaller batches based on locality, reducing the number of pages that need to be updated and journaled, thereby minimizing transaction journaling overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transaction journaling is used to ensure durability during snapshot deletion, then data integrity is maintained, but memory resource consumption increases and system performance decreases
Solution Approach 1:
The patent segments the transaction journaling process by introducing a pre-write buffer that separates the journaling operation from the main transaction flow. This allows the system to maintain durability guarantees while reducing the performance impact by buffering journal entries and processing them in a controlled manner rather than immediately blocking the main transaction path.
Solution Approach 2:
The pre-write buffer acts as an intermediary between the transaction system and the journaling mechanism. It receives transaction updates, prepares them for journaling in a controlled manner, and then writes them to the journal, thereby mediating the conflict between maintaining data integrity and preserving system performance during snapshot deletion operations.
2Reliability
If all dirty pages are journaled before flushing to disk, then transaction durability is ensured, but memory resources are consumed and system delays increase
Solution Approach 1:
The system performs preliminary action by pre-writing transaction updates to a buffer before they are journaled to disk. This preliminary buffering allows the system to prepare transaction data in memory without immediately committing it to the journal, thereby reducing the time critical path delays while still ensuring durability through the subsequent journaling process.
Solution Approach 2:
Instead of journaling all dirty pages immediately, the system applies partial action by selectively buffering and journaling only the necessary transaction updates in a controlled sequence. This partial journaling approach reduces the overall time burden while maintaining the durability guarantee for committed transactions.
3Productivity
If a large number of pages are updated and journaled during snapshot deletion, then complete snapshot removal is achieved, but memory resources are starved and performance deteriorates
Solution Approach 1:
The patent segments the snapshot deletion process into manageable units by processing pages through a pre-write buffer in controlled batches. This segmentation prevents the system from attempting to journal and flush all dirty pages simultaneously, thereby reducing peak memory resource consumption while still achieving complete snapshot removal through iterative processing.
Solution Approach 2:
The system dynamically adjusts its journaling and flushing behavior based on available memory resources and system state. The pre-write buffer allows the system to flexibly control when and how transaction updates are committed to the journal, enabling adaptive resource management that prevents memory starvation while ensuring complete snapshot deletion.
Data Source
AI summary
A method for deleting one or more snapshots using micro-batch processing is provided. The method includes receiving a request to delete the one or more snapshots, identifying one or more middle map extents exclusively owned by the one or more snapshots requested to be deleted, wherein metadata for the one or more snapshots is stored in one or more logical maps having logical map extents mapping logical block addresses (LBAs) to middle block addresses (MBAs) and a middle map having middle map extents mapping MBAs to physical block addresses (PBAs) of physical locations where data blocks are written, adding MBAs of the identified one or more middle map extents in a batch, determining a first micro-batch including a first subset of the MBAs in the batch, the first subset of MBAs being MBAs less than a first upper bound MBA, and using a first transaction to delete the middle map extents corresponding to the first subset of MBAs included in the first micro-batch.