Hash-Based Storage Deduplication via Modular Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems, particularly primary storage solutions, face challenges in scalability, performance, and efficiency due to their reliance on address-based mapping, which limits their ability to handle exponential data growth and results in increased costs, space, and power consumption, while also being unable to support content-based storage with its advantages such as deduplication and improved performance.
Innovation Solution
The implementation of a content-addressable storage (CAS) system using modular architecture with R-, C-, and D-modules, where data is mapped and managed based on its content through hashing, allowing for scalable, efficient storage and retrieval by avoiding duplicate writes and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If address-based mapping is used in storage systems, then data retrieval is straightforward and compatible with application requirements, but storage capacity efficiency deteriorates due to duplicate block data occupying multiple instances
Solution Approach 1:
The patent changes the mapping parameter from address-based to content-based (hash-based) mapping. Instead of storing data at predetermined addresses, the system computes a hash of the data content to determine storage location, enabling duplicate detection and deduplication while maintaining retrieval capability through the hash function.
2Quantity of substance
If content-based storage is implemented, then storage capacity efficiency improves through deduplication, but system complexity increases due to hashing and metadata management requirements
Solution Approach 1:
The patent segments the storage system into distinct functional modules: C-modules for data reception and hashing, D-modules for data storage and management, and R-modules for data retrieval. This segmentation distributes the complexity across specialized components, making the overall system more manageable despite the inherent complexity of content-based storage.
Solution Approach 2:
The patent introduces an address-to-hash table as an intermediary data structure that maps original data addresses to their corresponding hash values and storage locations. This intermediary simplifies the retrieval process by providing a direct lookup mechanism without requiring complex hash computation during data access.
3Device complexity
If address-based mapping is used, then system architecture is simpler and more direct, but performance deteriorates during exponential data growth due to increased duplicate writes
Solution Approach 1:
The patent performs preliminary hashing and duplicate detection before actual data writing occurs. By computing the hash of incoming data blocks and checking against existing hashes in the address-to-hash table, the system identifies duplicates in advance and avoids unnecessary write operations, improving performance during data growth phases.
4Adaptability or versatility
If content-based storage with modular architecture is implemented, then scalability improves for handling exponential data growth, but data integrity challenges increase due to distributed module coordination requirements
Solution Approach 1:
The patent implements feedback mechanisms where D-modules send acknowledgments to C-modules confirming successful data commitment, and R-modules verify data integrity during retrieval operations. This feedback loop ensures that distributed module coordination maintains data integrity even as the system scales to handle exponential data growth.
Data Source
AI summary
In one aspect, a method includes selecting a C-module; sending a write from a host to the selected C-module; selecting a D-module to commit a page related to the write; selecting a R-module to transmit data from the write to the target; writing the data from the write to target location; and writing the data to an address-to-hash table after acknowledgement that the data has been written to the target location and after the D-module acknowledges that the page is committed.