Metadata Deduplication for Storage Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face inefficiencies in memory usage due to storing duplicate data chunks during data management operations, leading to unnecessary memory waste as they store metadata for all data chunks, even when only a single copy is likely to be retained over time.
Innovation Solution
Implementing a post-processing metadata conversion process that deduplicates data chunks by identifying and removing entries for chunks not referenced by a threshold number of objects and modifying chunk identifiers from SHA-1 hash values to shorter representations, thereby reducing memory usage in the hot storage tier.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage system stores metadata for all data chunks to enable complete tracking and retrieval, then data retrieval reliability is improved, but memory usage increases significantly due to duplicate chunks
Solution Approach 1:
The patent extracts and removes duplicate data chunk entries from the metadata store, keeping only unique chunk references. This selective removal reduces memory usage while preserving the ability to retrieve all necessary data chunks through the remaining unique metadata entries.
Solution Approach 2:
The patent changes the state of metadata by transitioning from storing complete metadata for all chunks (including duplicates) to storing condensed metadata that references unique chunks. This parameter change in metadata representation reduces memory consumption while maintaining retrieval capability.
2Quantity of substance
If the storage system performs in-line deduplication to remove duplicate data chunks during ingestion, then storage utilization is improved, but processing time and computational resources increase
Solution Approach 1:
The patent performs deduplication as a preliminary action during the data ingestion process rather than as a separate subsequent operation. By identifying and removing duplicate chunks during ingestion, the system prepares the metadata store in advance, reducing future memory usage without requiring a separate deduplication pass.
Solution Approach 2:
The patent maintains continuous deduplication action during data ingestion operations, ensuring that duplicate chunks are identified and removed as they are being ingested. This continuous process eliminates the need for separate batch deduplication operations, optimizing both storage utilization and processing efficiency.
3Reliability
If the storage system uses SHA-1 hash values for chunk identifiers to ensure data integrity, then data security is improved, but memory usage increases due to the length of the identifiers
Solution Approach 1:
The patent extracts only the essential identifying information from the full SHA-1 hash values, storing condensed chunk identifiers in the metadata. This selective extraction reduces the memory footprint of metadata entries while maintaining the ability to uniquely identify and retrieve data chunks.
Solution Approach 2:
The patent changes the parameter of chunk identifier representation from full SHA-1 hash values (20 bytes) to condensed identifiers (4-8 bytes). This parameter change significantly reduces memory usage in the metadata store while preserving data integrity through the use of reference tables that map condensed identifiers to full hashes when needed.
Data Source
AI summary
Data associated with a source system is ingested. After the data is ingested, a post-processing metadata conversion process is performed including by selecting an entry of a chunk metadata data structure and determining that a data chunk associated with the selected entry is not referenced by at least a threshold number of objects. In response to determining that the data chunk associated with the selected entry is not referenced by at least the threshold number of objects, metadata of a tree data structure node corresponding to a chunk identifier associated with the data chunk is updated to store a reference to a chunk file storing the data chunk and the selected entry is removed from the chunk metadata data structure.