Snapshot Management in Distributed De-duplication Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in managing snapshots in distributed de-duplication systems due to complexity and scalability issues, particularly in reducing redundant data and optimizing storage and bandwidth usage.
Innovation Solution
The solution involves a method where storage controllers fragment and compress data, compute hash values, and use tenant keys to encrypt fragments, allowing for deduplication and efficient storage management by caching and storing only unique data, thereby minimizing storage and bandwidth consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored without deduplication, then storage capacity is consumed by redundant data, but implementing traditional deduplication requires complex distributed databases and deep integration into the storage system
Solution Approach 1:
The patent extracts the deduplication function from the core storage path by implementing it as a separate data processing pipeline. The deduplication operations are performed by dedicated processing components rather than being integrated into the main storage controllers, allowing the system to reduce redundant data storage without requiring deep integration into the storage system architecture.
Solution Approach 2:
The patent segments the deduplication process into distinct operational stages: data reception, hashing, duplicate detection, and selective storage. By dividing the deduplication function into separate processing steps handled by different components, the system achieves efficient redundancy reduction without requiring complex distributed databases or deep system integration.
2Reliability
If snapshots are managed in distributed de-duplication systems, then data integrity at point-in-time is preserved, but system complexity increases due to coordination across distributed nodes
Solution Approach 1:
The patent introduces a snapshot coordinator component that acts as an intermediary to manage snapshot operations across distributed nodes. This coordinator receives snapshot requests, coordinates the creation of consistent point-in-time copies across multiple storage nodes, and manages snapshot lifecycle operations, thereby preserving data integrity while reducing the complexity burden on individual storage controllers.
Data Source
AI summary
Methods, non-transitory computer readable media, and devices that more effectively manage snapshots by creating a namespace including files described by stored file recipe objects and each comprising fragments, directories described by metadata in the file recipe objects and each comprising one or more of the files, and snapshots described by stored snapshot request objects. Content of one of the directories are identified based on an identification of corresponding ones of the file recipe objects that existed within the namespace at a specified time. At least one of the files, included in the identified content and existing within the namespace at the specified time, is accessed in response to one or more received requests. A garbage collection operation is periodically performed to delete the recipe objects that are marked for deletion by tombstone objects and are unreferenced by any of the snapshots as determined based on the snapshot request objects.