Multi-key Encrypted Data Deduplication via Client-side Fingerprints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques fail to provide meaningful reduction when applied to encrypted data, as they break down due to unique encryption keys, and they do not ensure adequate data privacy between clients and storage systems.
Innovation Solution
A method involving sending key group information to a storage system, generating deduplication information with fingerprints, and encrypting data chunks using client data keys, where the decryption key is not available to the storage system, allowing deduplication across multiple encryption keys without exposing the keys to the storage system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data deduplication techniques are applied to encrypted data, then data privacy is maintained, but deduplication efficiency deteriorates because encryption keys prevent identification of duplicate data chunks
Solution Approach 1:
The system segments data into fixed-size chunks and generates unique fingerprints for each chunk. These fingerprints serve as identifiers that enable deduplication comparison without requiring decryption, thus maintaining privacy while enabling efficiency. The segmentation of data into chunks with independent fingerprints allows the storage system to identify duplicates across encrypted datasets.
Solution Approach 2:
The patent introduces fingerprints as an intermediary mechanism between the encrypted data and the deduplication process. Instead of comparing encrypted data directly (which is inefficient), the system compares fingerprints that represent the encrypted chunks. This intermediary enables the storage system to perform deduplication operations on encrypted data without accessing the encryption keys, resolving the contradiction between privacy and efficiency.
2Reliability
If full client-side encryption is used, then data privacy between client and storage system is maximized, but deduplication capability deteriorates because the storage system cannot access encryption keys
Solution Approach 1:
The system extracts the essential identifying characteristics of encrypted data chunks into fingerprints, which are then stored separately from the encrypted data. This extraction allows the storage system to perform deduplication operations on the fingerprint metadata without needing access to the encryption keys or the actual encrypted data, thereby maintaining full client-side encryption while restoring deduplication capability.
Solution Approach 2:
The patent transitions the deduplication operation from the data dimension (comparing encrypted bytes) to the metadata dimension (comparing fingerprints). By operating in this alternative dimension, the storage system can identify duplicates without compromising the encryption security model, enabling both strong privacy protection and effective deduplication.
3Ease of operation
If conventional BYOK encryption is implemented, then data reduction functions can be provided by the storage system, but data privacy between storage system and client deteriorates because the storage system has access to the client key
Solution Approach 1:
The system applies different quality requirements to different parts of the data processing pipeline. The storage system needs only the fingerprint information (local quality requirement) to perform deduplication, while the encryption keys remain locally retained by the client for security. This localized approach to information requirements enables the storage system to provide data reduction functions without compromising overall data privacy.
Data Source
AI summary
A computer-implemented method includes sending key group information to a storage system. The key group information includes keyID information for client data keys in the key group. The client data keys enable deduplication of data chunks encrypted in any of the client data keys in the key group. The method also includes generating deduplication information. The deduplication information includes fingerprints associated with chunks of client data. The method also includes encrypting the data chunks with one of the client data keys, wherein a corresponding decryption key for the encrypted data chunks is not available to the storage system. The method includes sending the deduplication information to the storage system for use in a deduplication process by the storage system and sending the encrypted data chunks to the storage system.