Encrypted Data Deduplication via Hash-Based Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data deduplication systems cannot effectively deduplicate encrypted data without compromising data security, as encryption prevents deduplication and distributing encryption keys is insecure, limiting the ability to achieve deduplication benefits, especially in non-secure network transfers and cloud storage.
Innovation Solution
The system deduplicates encrypted data by using a hash of the plaintext and associated metadata, allowing the backend server to identify and store unique ciphertext blocks without decrypting the data, even with key rollovers, by employing an initialization vector for encryption and transferring this information over a non-secure network for storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted prior to transfer and storage, then data security is improved, but deduplication capability deteriorates
Solution Approach 1:
The system segments the data handling process into distinct components: encryption is performed at the client side before transfer, while deduplication is performed at the storage system side on the encrypted data. This segmentation allows both encryption and deduplication to function independently without interfering with each other, resolving the contradiction between security and deduplication capability.
Solution Approach 2:
The patent introduces an intermediary mechanism - a deduplication index based on cryptographic hashes of encrypted data blocks. This intermediary allows the storage system to identify and eliminate duplicates without needing to decrypt the data or access encryption keys, thereby maintaining both security and deduplication functionality.
2Productivity
If encryption keys are distributed to the storage system to enable deduplication, then deduplication capability is improved, but data security deteriorates
Solution Approach 1:
The patent extracts the deduplication function from the encryption process. Instead of requiring the storage system to have encryption keys to perform deduplication, the system uses cryptographic hashes of the encrypted data blocks as identifiers. This extraction allows deduplication to occur on encrypted data without compromising security, as the storage system never needs to access the actual encryption keys.
3Productivity
If deduplication is performed on plaintext data, then deduplication efficiency is improved, but data security deteriorates
Solution Approach 1:
The patent inverts the traditional approach by performing deduplication on encrypted data rather than on plaintext data. The storage system computes cryptographic hashes of the encrypted data blocks and uses these hashes to identify duplicates. This inversion allows deduplication to occur after encryption, maintaining data security while achieving deduplication efficiency through hash-based comparison.
Data Source
AI summary
Plaintext data is encrypted to produce ciphertext which is transmitted along with a hash of the plaintext data and corresponding metadata comprising an initialization vector and information about the encryption key version used to encrypt the plaintext data to a backend storage system. The encrypted ciphertext is deduplicated at the backend storage system (without first decrypting it) using the hash and stored based upon the metadata.