Perfect Hash Vector for Deduplicated Storage Sanitization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in efficiently sanitizing deduplicated storage systems, particularly in managing chunk references to preserve live data while erasing dead data, due to high memory and I/O requirements, and the complexity of deduplication which makes individual file sanitization as difficult as sanitizing the entire file system.
Innovation Solution
The implementation of a sanitization method that uses a Perfect Hash Vector (PHVec) data structure to compactly represent fingerprints, allowing for efficient tracking and copying of live data chunks, and overwriting dead data chunks with patterns, while minimizing memory and I/O usage, and leveraging a copy bit vector to manage container operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional sanitization methods are used on deduplicated storage systems, then data security is improved, but memory and I/O requirements increase significantly
Solution Approach 1:
The patent segments the sanitization process into distinct phases: building a perfect hash function on a subset of fingerprints, identifying live data chunks through selective lookup, and erasing dead chunks. This segmentation allows the system to process large storage volumes by working with manageable subsets rather than requiring all data to be loaded into memory simultaneously, thus reducing memory and I/O requirements while maintaining data security.
Solution Approach 2:
The patent introduces a perfect hash function as an intermediary data structure that enables efficient identification of live data chunks without requiring comprehensive memory residency. This intermediary structure acts as a mediator between the storage system and the sanitization process, allowing the system to determine which chunks to preserve and which to erase with minimal memory and I/O overhead.
2Quantity of substance
If deduplication is implemented to reduce storage requirements, then storage efficiency is improved, but the complexity of sanitization increases
Solution Approach 1:
The patent leverages the existing deduplication infrastructure and fingerprint database to perform sanitization. The system uses the same fingerprint matching mechanisms already in place for deduplication operations, allowing the sanitization process to 'self-serve' using existing system resources and data structures rather than requiring separate complex tracking mechanisms.
Solution Approach 2:
The patent changes the operational parameters of the hash function by building it on a subset of fingerprints rather than requiring the complete fingerprint database to be resident in memory. This parameter change transforms the sanitization process from one requiring full system state awareness to one that can operate with partial information, thereby reducing complexity while maintaining effectiveness.
3Measurement precision
If secure hash values like SHA1 are used to identify data chunks, then data identification accuracy is improved, but memory requirements increase
Solution Approach 1:
The patent applies partial action by building the perfect hash function on a subset of fingerprints rather than requiring all fingerprints to be processed simultaneously. This allows the system to achieve sufficient identification accuracy for the current sanitization task without the excessive memory requirement of loading the entire fingerprint database into memory, thus balancing accuracy with resource constraints.
Data Source
AI summary
Techniques for sanitizing a storage system are described herein. In one embodiment, for each of fingerprints representing data chunks stored in a first container of the storage system, a lookup operation in a live bit vector based on the fingerprint is performed to determine whether a corresponding data chunk is live. In one embodiment, a bit in a copy bit vector corresponding to the data chunk is populated based on the lookup operation. In one embodiment, after all of the bits corresponding to the data chunks of the first container have been populated in the CBV, data chunks represented by the CBV are copied from the first container to a second container, and records of the data chunks in the first container are erased.