Snapshot Storage Space Optimization for Data Durability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in providing guaranteed durability without unnecessarily storing excess data or parity copies, and lack customer-centric options for durability settings and snapshot optimization, leading to potential data loss and increased costs.
Innovation Solution
A data storage system that allows customers to specify durability requirements and enables snapshot optimization, where data is stored with fewer copies or parity partitions if a complete redundant copy is stored in a separate system, optimizing storage resources while maintaining durability through metadata management and erasure encoding schemes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored with multiple copies or parity partitions to ensure durability, then data reliability is improved, but storage resource consumption increases
Solution Approach 1:
The system applies different storage strategies to different data chunks based on their snapshot status. Clean chunks (those with complete snapshots) use reduced redundancy storage, while dirty chunks (modified since snapshot) use full redundancy. This local differentiation resolves the contradiction by optimizing storage resources for each data portion according to its actual durability needs.
Solution Approach 2:
The system dynamically adjusts the quantity of stored copies or parity partitions based on the current state of data and snapshots. When data is modified, the system transitions from snapshot-based storage to full redundancy storage for affected chunks. This dynamic adaptation allows the system to maintain durability guarantees while minimizing storage resource consumption at any given moment.
2Reliability
If complete redundant copies are stored in separate storage systems, then data security is improved, but storage costs increase
Solution Approach 1:
The system creates snapshots as complete redundant copies in separate storage systems, but only for clean chunks that haven't been modified. This copying strategy ensures data security for unchanged portions while avoiding the cost of continuously maintaining full copies of all data. The snapshot mechanism provides security at lower cost compared to maintaining complete real-time replicas.
Solution Approach 2:
The system applies snapshot-based redundancy only to clean chunks while using different strategies for dirty chunks. This local quality approach optimizes the balance between security and cost by using snapshot redundancy where it provides maximum benefit (unchanged data) while avoiding unnecessary costs for modified data that requires full redundancy anyway.
3Quantity of substance
If snapshot optimization is enabled to reduce storage resources, then storage efficiency is improved, but data retrieval time may increase
Solution Approach 1:
The system performs preliminary actions by maintaining snapshots of clean chunks in advance. When data needs to be retrieved, if the chunk is clean, the system can quickly restore from the pre-existing snapshot without needing to reconstruct from parity or retrieve from multiple sources. This preliminary preparation reduces retrieval time while maintaining storage efficiency.
Solution Approach 2:
The system dynamically selects retrieval strategies based on chunk status. For clean chunks, it uses fast snapshot restoration; for dirty chunks, it uses full redundancy retrieval. This dynamic approach ensures optimal retrieval performance for each data portion while maintaining overall storage efficiency through snapshot optimization.
Data Source
AI summary
A data storage system includes a head node and mass storage devices. The head node is configured to store volume data and flush volume data to the mass storage devices. Additionally, the head node is configured to determine a quantity of data partitions and/or parity partitions to store for a chunk of volume data being flushed to the mass storage devices in order to satisfy a durability guarantee. For chunks of data for which complete copies are also stored in an additional data storage system, the head node is configured to reduce the quantity of data partitions and/or parity partitions stored such that required storage space is reduced while still ensuring that the durability guarantee is satisfied.