Segment Deduplication via Pre-Encryption Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication storage systems face inefficiencies when trying to identify identical segments after encryption or compression, as these processes typically prevent effective segment identification, thereby reducing the system's efficiency.
Innovation Solution
A system that breaks data streams, blocks, or files into segments, encrypts and compresses them, and then determines if they have been previously stored, allowing for efficient deduplication by using identifiers like digital fingerprints or hash values to check for duplicates, while also enabling decryption and decompression for data reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data segments are encrypted to ensure security, then security is improved, but the ability to identify identical segments for deduplication is lost
Solution Approach 1:
The system calculates a digital fingerprint (hash value) of the plaintext segment before encryption. This preliminary action allows the system to identify identical segments based on their content before encryption occurs, enabling deduplication to function effectively even though the actual stored data is encrypted. The fingerprint serves as a preliminary identifier that preserves deduplication capability while the encrypted form ensures security.
2Volume of stationary object
If data segments are compressed to save space, then storage efficiency is improved, but the ability to identify identical segments is lost
Solution Approach 1:
The system calculates a digital fingerprint (hash value) of the plaintext segment before compression. This preliminary action allows the system to identify identical segments based on their content before compression occurs, enabling deduplication to function effectively even though the actual stored data is compressed. The fingerprint serves as a preliminary identifier that preserves deduplication capability while the compressed form saves storage space.
3Reliability
If both encryption and compression are applied to segments, then security and storage efficiency are improved, but the ability to identify identical segments is completely lost
Solution Approach 1:
The system calculates a digital fingerprint (hash value) of the plaintext segment before both encryption and compression operations. This preliminary action allows the system to identify identical segments based on their content before any transformation occurs, enabling deduplication to function effectively even though the actual stored data undergoes both encryption and compression. The fingerprint serves as a preliminary identifier that preserves deduplication capability while the combined encryption and compression provide both security and space efficiency.
Data Source
AI summary
A system for storing encrypted compressed data comprises a processor and a memory. The processor is configured to determine whether an encrypted compressed segment has been previously stored. The encrypted compressed segment was determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments. The processor is further configured to store the encrypted compressed segment in the event that the encrypted compressed segment has not been previously stored. The memory is coupled to the processor and configured to provide the processor with instructions.