Snapshot Hierarchy and Segment GC for Distributed Storage Rollback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for creating snapshots of storage volumes are inefficient in managing large-scale data distribution and fail to effectively handle write requests and garbage collection, leading to potential data corruption and complexity in rolling back to previous states.
Innovation Solution
The proposed solution involves a network environment with a storage manager coordinating snapshot creation, quiescing compute nodes to suppress write requests, and performing garbage collection to ensure data integrity, while allowing for efficient snapshot management and rollback capabilities across multiple storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If snapshots are created in a distributed storage system, then data versioning and rollback capability are improved, but system complexity and risk of data corruption increase
Solution Approach 1:
The system divides the storage volume into multiple segments and creates snapshots at the segment level. Each segment can be independently versioned and managed, reducing the overall complexity of snapshot management in distributed systems while maintaining rollback capability.
Solution Approach 2:
The patent introduces an intermediary mechanism (snapshot manager) that coordinates between compute nodes and storage nodes during snapshot creation. This intermediary handles the complexity of data corruption prevention and rollback coordination, isolating these complexities from the core storage and compute operations.
2Reliability
If write requests are suppressed during snapshot creation, then data integrity is improved, but system productivity and response time deteriorate
Solution Approach 1:
The system performs preliminary actions by identifying and isolating segments that will be modified during snapshot creation. Compute nodes are notified in advance about which segments will be quiesced, allowing them to prepare alternative write paths or buffer operations, thereby maintaining productivity while ensuring integrity.
Solution Approach 2:
The snapshot creation process dynamically adjusts write suppression based on segment-level operations. Instead of globally suppressing all writes, the system dynamically allows writes to segments that are not part of the snapshot, maintaining system productivity while ensuring data integrity for the specific segments being snapshotted.
3Loss of substance
If garbage collection is performed during snapshot management, then storage efficiency is improved, but risk of data corruption and system complexity increase
Solution Approach 1:
The garbage collection mechanism operates autonomously by monitoring segment references across snapshots. When segments are no longer referenced by any active snapshot, the system automatically reclaims them without requiring manual intervention or complex coordination, improving storage efficiency while maintaining data integrity through self-verification.
Solution Approach 2:
The system implements feedback mechanisms where garbage collection operations continuously monitor snapshot references and adjust collection timing accordingly. Before collecting any segment, the system verifies it is not needed by any active snapshot, providing feedback that prevents data corruption while optimizing storage efficiency.
Data Source
AI summary
A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. The snapshots may be represented by a storage manager in a hierarchy. Deleted snapshots may be flagged as such in the hierarchy and deletion may be implemented only in memory on a storage node, which is then restored from the hierarchy in the event of a crash. A snapshot is removed from the hierarchy when all segments previously are freed by garbage collection. A hybrid storage node may perform both computing and storage services. Data may be written with tags indicating encoding protocols used to encode the data.