Incremental Checkpointing Data Units Storage Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficient checkpointing in data processing systems with large numbers of data units is challenging due to high data transfer costs and storage overhead, especially when only a small fraction of data units change during each checkpoint interval, requiring a balance between data transfer time and recovery complexity.
Innovation Solution
The system implements an incremental checkpointing scheme where both new and old data units are stored in the same checkpoint file, with old copies gradually consolidated and removed, using pointers to manage access and storage, allowing for efficient data transfer and reduced storage costs by limiting extra work during checkpoint intervals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data units are checkpointed at each interval, then recovery reliability is improved, but data transfer cost and storage overhead increase significantly
Solution Approach 1:
The patent extracts only the changed data units from the working data set and stores them in the checkpoint file, rather than checkpointing all data units. This extraction approach reduces data transfer cost while maintaining recovery reliability, as only the modified portions need to be transferred and stored.
Solution Approach 2:
The patent applies partial action by checkpointing only a subset of data units (those that have changed) rather than the complete set. This partial checkpointing strategy reduces storage overhead and transfer costs while still ensuring reliable recovery of the latest state.
2Quantity of substance
If old copies of data units are removed from checkpoint files, then storage overhead is reduced, but recovery complexity increases
Solution Approach 1:
The patent uses a feedback mechanism where the system tracks which data units have been accessed or modified since the last checkpoint. This feedback information guides the selective removal of old copies from checkpoint files, ensuring that only obsolete data is removed while maintaining recovery simplicity.
Solution Approach 2:
The checkpointing system performs self-service by automatically managing the lifecycle of data unit copies in checkpoint files. It autonomously determines which old copies to remove based on access patterns, reducing manual intervention while maintaining low recovery complexity.
3Loss of energy
If incremental checkpointing is implemented, then data transfer cost is reduced, but the system complexity increases due to tracking changes
Solution Approach 1:
The patent introduces an intermediary mechanism (access tracking system) that mediates between the working data and the checkpointing process. This intermediary tracks which data units have been accessed or modified, enabling incremental checkpointing without significantly increasing overall system complexity.
Data Source
Figure 1
Figure 2A~2B
Figure 3A
AI summary
A memory module (108) stores working data (114) that includes data units. A storage system (116) stores recovery data (120) that includes sets of one or more data units. Transferring data units between the memory module and the storage system includes: maintaining an order among the data units included in the working data, the order defining a first contiguous portion and a second contiguous portion; and, for each of multiple time intervals (202A, 202B), identifying any data units accessed from the working data during the time interval, and adding to the recovery data a set of two or more data units including: one or more data units from the first contiguous portion including any accessed data units, and one or more data units from the second contiguous portion including at least one data unit that has been previously added to the recovery data.