Storage Synchronization Using Log Files and Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file and storage systems face inefficiencies in reverting data to a previous state due to the complexity and slowness of recopying entire data sets from replica systems, especially in large databases where log files grow unmanageable and require frequent snapshots to track changes efficiently.
Innovation Solution
A system utilizing deduplication, log files, and snapshots to identify and transfer only changed data segments between snapshots, using fingerprint comparisons to reconstruct data efficiently and reduce the need for full data set copying, allowing for 'lazy' retrieval of unchanged data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data set is recopied from replica system to restore data to previous state, then data recovery is achieved, but the process becomes very complex and slow
Solution Approach 1:
The patent divides the data set into individual data blocks and tracks changes at the block level using log files. Instead of copying the entire data set, only the changed blocks need to be transferred and rewritten to the replica system, significantly reducing the recovery time and complexity.
Solution Approach 2:
The patent extracts only the necessary changed data blocks from the replica system for restoration, rather than extracting the entire data set. The log files are used to identify and extract only the minimal set of blocks that changed since the last known good state.
2Productivity
If log files are used to track data changes, then data restoration efficiency is improved, but log files grow unmanageably large
Solution Approach 1:
The patent performs preliminary actions by creating snapshots of the data set at regular intervals or when significant changes occur. These snapshots serve as checkpoints that limit the growth of log files, as only changes between snapshots need to be tracked in the log files rather than all changes since the beginning of time.
Solution Approach 2:
The patent segments the log file management by using multiple snapshots as delimiters. The log files track changes between snapshot points, effectively dividing the large log into manageable segments that are easier to manage and restore from.
3Measurement precision
If frequent snapshots are created to track changes, then data recovery precision is improved, but storage space requirements increase
Solution Approach 1:
The patent applies local quality by creating snapshots only when necessary (e.g., after significant data changes or at scheduled intervals) rather than continuously. This allows for precise change tracking at critical points while minimizing the total storage space required for snapshot maintenance.
Solution Approach 2:
The patent implements a strategy where older snapshots are discarded or pruned when they are no longer needed for recovery purposes. The log files track changes forward from snapshots, allowing the system to recover data to any snapshot point while managing storage by removing obsolete snapshots that cannot be recovered from.
Data Source
AI summary
A system for processing data comprises a deduplicating system, an interface, and a processor. The deduplicating system stores a copy of data stored in a data storage system by storing a set of segments that is able to reconstruct the data stored in the data storage system. The data storage system has a stored log file. The stored log file stores a data change and an associated time for the data change. The interface receives an indication to revert data stored in the data storage system to a state at a snapshot time. The processor is configured to determine a first subset of the data stored in the data storage system that has changed since a prior snapshot using the stored log file and to determine a second subset of the data stored in the data storage system that has changed between the prior snapshot and the snapshot time using a first list of fingerprints associated with the prior snapshot and a second list of fingerprints associated with the snapshot time.