Distributed Storage De-duplication via Snapshot Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage and computation systems face inefficiencies in data management due to the lack of effective methods for eliminating duplicated data, leading to increased storage requirements and processing burdens, particularly in append-only storage schemes and garbage collection processes.
Innovation Solution
The implementation of a method that coordinates snapshot creation across storage and compute nodes, utilizing a storage manager to quiesce write requests, create snapshots, manage segment allocation, and perform garbage collection, thereby identifying and eliminating duplicate data segments across multiple storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If de-duplication is not implemented in distributed storage systems, then data can be stored and accessed without complex coordination protocols, but storage space is wasted due to duplicate data segments across multiple nodes
Solution Approach 1:
The patent segments data into fixed-size segments with unique identifiers, allowing individual segment tracking and de-duplication across distributed nodes. Each segment can be independently managed, referenced, and deduplicated without requiring complex coordination of entire data structures.
Solution Approach 2:
The patent uses copy-on-write semantics where snapshot references are copied rather than the actual data. When a segment is de-duplicated, the reference is updated to point to a single canonical copy, eliminating duplicates while maintaining snapshot integrity through lightweight reference management.
2Reliability
If snapshot creation is coordinated across distributed nodes without quiescing writes, then system availability is maintained, but snapshot consistency cannot be guaranteed
Solution Approach 1:
The patent performs preliminary quiescing of write requests to specific storage nodes before creating snapshots. This preliminary action ensures that no writes occur to segments being snapshotted, guaranteeing consistency without requiring system-wide shutdown or prolonged unavailability.
Solution Approach 2:
The patent implements dynamic snapshot creation where the quiescing scope is adjusted based on which segments need to be snapshotted. Only the minimal necessary nodes and segments are quiesced temporarily, while the rest of the system continues operating, providing dynamic balance between consistency and availability.
3Productivity
If duplicate data segments are not eliminated, then write operations can proceed without additional processing, but write amplification increases and garbage collection becomes more burdensome
Solution Approach 1:
The patent merges duplicate data segments by identifying segments with identical content across different nodes and consolidating them into a single canonical segment. References to duplicate segments are updated to point to the canonical segment, reducing write amplification by eliminating redundant write operations to duplicate data.
Solution Approach 2:
The patent discards duplicate data segments after verifying their redundancy through content comparison. By identifying and removing duplicate segments while maintaining references to the canonical copy, the system recovers storage space and reduces garbage collection burden without affecting data accessibility.
Data Source
AI summary
A new snapshot of a storage volume is created by suppressing write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. An orchestration layer implements a bundled application that is provisioned with storage volumes and containers. A snapshot of the application may be created and used to rollback or clone the application. De-duplication may be performed by creating a signature map and identifying duplicated blocks. Blocks of segments containing duplicated blocks are copied to pool segments and metadata of those segments of the same logical storage unit may be consolidated to pool metadata segments. The identification of duplicate blocks may be performed in a cloud computing platform.