File Chunk Hash Deduplication in SDDC Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data center solutions face challenges in efficiently managing and duplicating data across software-defined data centers (SDDCs) for disaster recovery, particularly in reducing storage space and improving cache efficiency during deduplication processes.
Innovation Solution
The implementation of file chunk hashes using a secure hash algorithm (SHA-256) and a two-map approach (hash map and chunk map) within a content-addressable storage system, which allows for deduplication by associating chunk identifiers with hashes and segments, providing spatial locality and reducing cache misses, and merging adjacent tuples for efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data copying methods are used for disaster recovery in SDDCs, then data can be stored and recovered, but storage space is wasted due to duplication of identical data across multiple SDDCs
Solution Approach 1:
The patent implements deduplication by storing only unique data chunks and creating references/cursors to them, rather than copying actual data. When data needs to be accessed from multiple SDDCs, the system stores a reference (cursor) pointing to the original chunk location and retrieves data on-demand, eliminating redundant storage while maintaining disaster recovery capability.
Solution Approach 2:
The patent introduces an intermediary cursor mechanism that acts as a reference between the logical data structure and physical storage location. This cursor contains metadata (chunk ID, offset, length) that enables the system to access original data from a single storage location across multiple SDDCs without duplicating the actual data chunks.
2Quantity of substance
If data is deduplicated across SDDCs using traditional methods, then storage space is reduced, but cache efficiency deteriorates due to non-contiguous data access patterns
Solution Approach 1:
The patent segments data into fixed-size chunks and assigns each chunk a unique identifier. This segmentation allows the system to manage deduplicated data efficiently by working with discrete, addressable units rather than large contiguous blocks, enabling better cache utilization through targeted access patterns.
Solution Approach 2:
The patent transforms the storage access model from spatial contiguity to dimensional indexing by introducing chunk IDs as an additional dimension. Instead of relying on physical contiguity for cache efficiency, the system uses logical indexing (chunk ID, offset, length) to locate and retrieve data, enabling efficient cache utilization even with deduplicated, non-contiguous storage.
Data Source
AI summary
The present disclosure is related to methods, systems, and machine-readable media for supporting deduplication in file storage using file chunk hashes. A hash of a chunk of a log segment can be received from a software defined data center. A chunk identifier can be associated with the hash in a hash map that stores associations between sequentially-allocated chunk identifiers and hashes. The chunk identifier can be associated with a logical address corresponding to the chunk of the log segment in a logical map that stores associations between the sequentially-allocated chunk identifiers and logical addresses. A search of the hash map can be performed to determine if the chunk is a duplicate, and the chunk can be deduplicated responsive to a determination that the chunk is a duplicate.