Data Deduplication Encryption Key Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In untrusted third-party storage systems, user-specific encryption keys prevent data deduplication since encrypted files differ among clients, limiting storage efficiency.
Innovation Solution
Generating an encryption key and identifier based on a hash of the data, allowing detection of existing encrypted copies and selective encryption to maintain security and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If user-specific encryption keys are used to encrypt client data, then data security is improved, but data deduplication capability deteriorates
Solution Approach 1:
The encryption process is segmented into two independent parts: (1) client-specific encryption using user-specific keys for security, and (2) content-based identifier generation using hashing for deduplication. This segmentation allows both security and deduplication to function simultaneously without interference.
Solution Approach 2:
A content-based identifier (hash value) is introduced as an intermediary between the encrypted data and the deduplication mechanism. This identifier serves as a mediator that enables deduplication operations without requiring access to or knowledge of the user-specific encryption keys, thus preserving both security and deduplication functionality.
2Reliability
If each client encrypts data with different encryption keys, then confidentiality between clients is improved, but storage efficiency deteriorates
Solution Approach 1:
The storage system segments data identification into two independent layers: (1) cryptographic encryption layer using user-specific keys for confidentiality, and (2) deduplication layer using content-based hash identifiers for space optimization. This dual-layer approach maintains confidentiality while enabling storage efficiency.
Solution Approach 2:
The system changes the parameter used for data identification from encryption-key-dependent to content-dependent. By using hash values of the original data as identifiers rather than relying on encrypted forms, the system enables storage efficiency improvements while maintaining confidentiality through separate encryption mechanisms.
3Reliability
If encrypted versions of the same file differ among clients, then individual client security is improved, but deduplication opportunities are lost
Solution Approach 1:
A content-based identifier (hash value) serves as an intermediary that bridges individual client security and deduplication efficiency. This identifier is derived from the original data before encryption and remains consistent across clients, enabling deduplication without compromising the security provided by user-specific encryption keys.
Solution Approach 2:
The system changes the basis for identifying duplicate data from encryption-specific parameters to content-based parameters. By using hash values of the original unencrypted data as the identification parameter, the system enables deduplication while maintaining individual client security through separate encryption processes.
Data Source
AI summary
Various methods and systems for performing data deduplication when encrypting data for storage on an untrusted storage system are disclosed. One method involves generating an encryption key for use in encrypting data and generating an identifier for the data. Generation of the encryption key is based upon a hash of the data to be encrypted. Similarly, generation of the identifier is based upon the hash of the data to be generated. For example, the identifier can be generated by hashing the encryption key. The method then involves detecting whether an encrypted copy of the data is already stored by a storage system, based upon the identifier, and selectively encrypting the data, based upon whether the encrypted copy of the data is already stored by the storage system.