Distributed Deduplication of Encrypted Chunks via Independent Key Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud storage systems face challenges in efficiently storing and transmitting encrypted data chunks while maintaining the ability to identify identical chunks for deduplication, as existing solutions either compromise data security or rely on predictable encryption keys that can be exploited.
Innovation Solution
A distributed deduplication system that generates random encryption keys independently for each chunk, allowing for secure end-to-end encryption and identification of identical chunks even when encrypted with different keys, while preventing key predictability and ensuring system security if one chunk's encryption is compromised.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deterministic encryption is used to enable deduplication of encrypted chunks, then deduplication efficiency is improved, but data security deteriorates because identical plaintext chunks produce identical ciphertext chunks which can be exploited
Solution Approach 1:
The encryption process is segmented into two independent parts: (1) encryption of the chunk payload using a deterministic key derived from the plaintext, and (2) encryption of the chunk key using a random server key. This segmentation allows the payload encryption to be deterministic for deduplication while the key encryption provides security by preventing plaintext recovery.
Solution Approach 2:
A server-side random key acts as an intermediary between the client's deterministic encryption and the stored ciphertext. The server encrypts the client-provided chunk key with its own random key, creating a layered encryption scheme where the intermediary key provides security while allowing the underlying deterministic encryption to enable deduplication.
2Reliability
If random encryption keys are generated for each chunk to improve data security, then security is improved, but the ability to identify identical chunks for deduplication deteriorates
Solution Approach 1:
The client performs preliminary action by generating a deterministic chunk key from the plaintext chunk before encryption. This preliminary key derivation ensures that identical plaintext chunks produce identical keys, enabling the server to identify duplicates through key comparison without needing to decrypt the actual chunk data.
Solution Approach 2:
The encryption scheme changes the parameter of key generation from random to deterministic (derived from plaintext). This parameter change allows identical plaintext to produce identical encryption keys, enabling the server to perform deduplication by comparing keys rather than comparing encrypted data, thus maintaining security while enabling deduplication.
3Productivity
If encryption keys are derived from chunk payloads to enable deduplication, then deduplication capability is improved, but security deteriorates because the encryption becomes predictable and exploitable
Solution Approach 1:
The solution moves the predictability from the encryption key space to a separate key derivation space. The deterministic relationship between plaintext and chunk key exists in one dimension (for deduplication), while the actual encryption operates in another dimension using random server keys, eliminating the security vulnerability of predictable encryption keys.
Solution Approach 2:
The server's random key acts as an intermediary that breaks the direct predictable relationship between plaintext and encryption key. While the chunk key is deterministically derived from plaintext, the actual encryption uses the intermediary server key, which is random and unpredictable, thus eliminating key predictability attacks while preserving deduplication through key comparison.
Data Source
AI summary
The present disclosure relates to an advantageous system and related methods for distributed deduplication of encrypted chunks. One embodiment relates to a method for storing encrypted chunks in which an encryption key is generated independently from a chunk payload. With this method, two encrypted chunks are identifiable as having identical chunk payloads even when the chunk payloads are encrypted with different encryption keys. Other embodiments, aspects and features are also disclosed.