Storage Array Snapshot Deduplication via Content-Addressable Hash Trees
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems, particularly primary storage solutions, face challenges in implementing content-based storage schemes due to intensive computational load, block fragmentation, loss of sequential data block location, and architectural difficulties in distributing tasks across processing and storage elements, leading to performance degradation and inefficiencies in handling exponential data growth.
Innovation Solution
A method and system that utilize a file system on a storage array to create snapshots and copy them into a file, employing hash trees and deduplication techniques to manage metadata efficiently, ensuring data integrity and long-term retention while minimizing metadata storage and optimizing performance by using content-addressable storage arrays with separate control and data planes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If content-based storage scheme is implemented, then storage capacity efficiency is improved, but computational load increases
Solution Approach 1:
The system segments data into fixed-size blocks and uses content-based addressing at the block level. Each block is independently hashed and stored, allowing parallel processing of multiple blocks without requiring intensive whole-file computation. This segmentation enables content-based storage efficiency while distributing computational load across many small, manageable units.
Solution Approach 2:
The patent introduces a content addressable storage (CAS) layer as an intermediary between traditional address-based storage and the data blocks. This CAS layer handles the content-based mapping and deduplication logic, separating the computational intensive hashing and deduplication tasks from the core storage operations. The intermediary manages metadata and block references, reducing the computational burden on the main storage system.
2Measurement precision
If large blocks are broken into smaller blocks for content addressing, then content addressing granularity is improved, but performance degrades due to block fragmentation
Solution Approach 1:
The system applies different block sizes and handling strategies to different data types and access patterns. Frequently accessed data uses larger blocks to minimize fragmentation overhead, while data requiring fine-grained content addressing uses smaller blocks. This local optimization maintains content addressing granularity where needed while preserving performance for performance-critical operations.
Solution Approach 2:
The system performs preliminary hashing and content verification on incoming data blocks before they are stored and fragmented. By pre-computing content hashes and establishing the content-based address mapping in advance, the system avoids repeated computation during fragmentation and retrieval operations. This preliminary action reduces the performance impact of subsequent block fragmentation.
3Speed
If address-based mapping is used, then sequential data block location is maintained, but storage capacity efficiency decreases due to duplicate data
Solution Approach 1:
The patent merges address-based and content-based mapping approaches into a hybrid system. The content-based addressing provides deduplication and capacity efficiency, while a separate address translation layer maintains sequential access patterns by mapping logical sequential addresses to physical content-based block locations. This merging allows the system to achieve both storage efficiency and sequential access performance simultaneously.
Data Source
AI summary
A method, system and computer product, comprising creating a file system on a LUN of a storage array, taking a snapshot of a second LUN at a first point in time, stored on the storage array, creating a file in the file system, and copying the snapshot into the file.