Primary Storage Backup Overhead Reduction via Dirty Block Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems incur significant overheads due to copy-on-write operations during incremental backups, which are inefficient as they require duplicating data even for unchanged blocks, affecting performance and resource utilization.
Innovation Solution
Implementing a method where dirty data blocks are identified and transferred in a non-sequential order based on access assessment, with copy-on-write or redirect-on-write operations only applied to blocks that need data integrity preservation, and marking blocks as 'normal' after transfer to avoid unnecessary duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If copy-on-write operations are performed during incremental backups to ensure data integrity, then data reliability is improved, but I/O operations and storage overhead increase significantly
Solution Approach 1:
The system performs preliminary identification of dirty data blocks before the backup process begins. By pre-marking blocks that have been modified since the last backup, the system avoids performing copy-on-write operations on clean blocks during the backup process, thereby reducing I/O overhead while maintaining data integrity for blocks that actually need it.
Solution Approach 2:
The system applies copy-on-write operations selectively only to dirty data blocks that require data integrity preservation, rather than uniformly to all blocks. This localized application of the copy-on-write mechanism reduces unnecessary I/O operations on clean blocks while ensuring reliability where needed.
2Reliability
If all data blocks are copied during incremental backup to ensure completeness, then backup reliability is improved, but transfer time and network bandwidth consumption increase
Solution Approach 1:
The system extracts and identifies only the dirty data blocks that have been modified since the last backup, separating them from the clean blocks. By transferring only these identified dirty blocks rather than all blocks, the system reduces transfer time and network bandwidth consumption while maintaining backup completeness through the use of change trackers that ensure no modified blocks are missed.
3Stability of the object's composition
If data blocks are transferred in sequential order during backup, then data consistency is improved, but transfer speed decreases due to I/O wait times
Solution Approach 1:
The system dynamically determines the transfer order of dirty data blocks based on their physical location and access patterns on the storage device. Instead of rigid sequential transfer, the system can reorder blocks to minimize I/O seek time and maximize transfer efficiency, adapting the transfer strategy to the actual state of the storage medium while maintaining data consistency through proper tracking.
Data Source
AI summary
Techniques for reducing overheads of primary storage while transferring modified data are described. According to one embodiment, in response to a request at a primary storage for a backup of a plurality of data blocks, a snapshot of the plurality of data blocks is captured, wherein the snapshot represents a consistent state of the plurality of data blocks. Then a set of data block of the captured snapshot that are to be copied to a secondary storage is identified and the identified set of data blocks are transferred to a secondary storage over a network. In response to receiving data to be written to a first data block at the first primary storage during the transfer, current contents of the first data block is preserved if the first data block is in identified set of data blocks, and the received data is committed to the first data block.