Content Addressable Storage System Block-Level Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Content addressable storage systems face inefficiencies due to slow write operations caused by the need to compute hashes for determining duplicate data at the file level, and existing solutions like the Carnegie Mellon University's NASD system suffer from performance issues and unreliable public networks.
Innovation Solution
A content addressable storage system with multiple storage bricks organized in a RAID configuration, where each brick performs content addressable storage computations in parallel, allowing for efficient aggregation of resources and elimination of duplicate data blocks through hashing, and using a meta-directory to manage data containers and user data across local and remote storage arrays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If content addressable storage computation is performed at the file level to determine duplicate data, then data deduplication is achieved, but write operations become slow due to substantial computation time
Solution Approach 1:
The patent segments the content addressable storage computation from the file level to the block level. Each data block is independently hashed and checked for duplicates rather than computing hashes at the entire file level. This segmentation reduces the computational burden per operation while maintaining deduplication effectiveness, as each block can be processed independently and in parallel.
Solution Approach 2:
The system performs preliminary hashing of data blocks as they are received, computing hash values immediately upon block arrival rather than waiting for complete file processing. This preliminary action enables early duplicate detection and allows the system to prepare deduplication decisions before full write operations complete, improving overall write performance.
2Productivity
If multiple storage bricks are used to aggregate resources for parallel computation, then write performance improves, but system complexity increases
Solution Approach 1:
The system divides storage resources into multiple independent storage bricks, each capable of handling content addressable storage computations autonomously. This segmentation enables parallel processing of data blocks across different bricks, improving write performance while keeping each individual brick relatively simple in design.
Solution Approach 2:
Each storage brick is designed as a universal unit that can perform multiple functions: data reception, hash computation, duplicate detection, and data storage. This multi-functionality reduces the need for specialized components and simplifies the overall system architecture while enabling parallel operations across multiple bricks.
3Quantity of substance
If hash computation is performed on every data block to eliminate duplicates, then storage space is optimized, but computation time increases substantially
Solution Approach 1:
The patent applies segmentation by computing hashes at the block level rather than file level, creating smaller, independent units of computation. This reduces the time required for each hash computation and enables parallel processing, while still achieving comprehensive deduplication across the entire storage system.
Solution Approach 2:
The system performs hash computation continuously as data blocks arrive, rather than batching all computations after data collection. This continuous useful action eliminates idle time and ensures that deduplication decisions are made immediately, optimizing storage space utilization without substantial delays.
Data Source
AI summary
A content addressable storage system (CASS) comprises a storage system coupled to one or more content addressable storage array elements (CASAEs). The storage system is coupled to storage resources of a local storage array configured to store a high level meta-directory that describes data containers, e.g., directories, files and logical units (luns), served by the CASS. In addition, each CASAE is coupled to storage resources of a remote storage array configured to store user data (data blocks) of the data containers served by the CASS. The CASAE is configured to perform a content addressable storage computation on each data block written to its storage resources in order to prevent storage of two or more blocks with the same data. Notably, the CASAE interacts with its associated remote storage array to form a CASAE “storage brick”.