Encrypted Data Deduplication Using Pre-Encryption Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication methods are limited in handling encrypted data, especially for 'in-transit' data, as they typically require the same encryption key for deduplication, which restricts the effectiveness of deduplication when different encryption keys are used.
Innovation Solution
The method generates a hash of the data before encryption and links it to a key index, allowing deduplication at the destination even with different encryption keys by creating associations between identifiers and stored encrypted data, preventing duplicate storage of identical data in different encrypted forms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted before storage using different encryption keys, then data security is improved, but deduplication effectiveness deteriorates because identical data produces different encrypted forms
Solution Approach 1:
The patent computes the hash of the plaintext data before encryption occurs. This preliminary hashing allows the system to identify duplicate data contents even when different encryption keys are used, because the hash is derived from the original unencrypted data and remains consistent regardless of which encryption key is applied later.
Solution Approach 2:
The patent introduces a hash value as an intermediary representation of the data. Instead of comparing encrypted data directly (which would fail for different keys), the system uses the hash as a mediator to identify duplicates, then uses metadata and key indices to manage the relationship between the hash, encrypted versions, and their corresponding encryption keys.
2Reliability
If multiple encryption keys are used for the same data, then data security and access control are improved, but storage efficiency deteriorates due to duplicate encrypted versions
Solution Approach 1:
The patent merges multiple encrypted versions of the same data into a single storage location by using a common hash identifier. The system stores one copy of the encrypted data and manages multiple key indices that can access it, thereby combining what would otherwise be separate storage resources into a unified structure.
Solution Approach 2:
The patent creates lightweight copies in the form of metadata entries and key indices rather than copying the actual encrypted data. Each access request with a different key is handled by creating or retrieving a metadata record that links the key index to the stored encrypted data, avoiding redundant storage of the heavy encrypted payload.
3Quantity of substance
If deduplication is performed on encrypted data, then storage space is reduced, but implementation complexity increases due to key management requirements
Solution Approach 1:
The patent segments the deduplication system into distinct functional components: a hashing component that processes plaintext, a metadata management component that tracks key indices and encrypted data locations, and a retrieval component that handles key-based access. This segmentation allows each component to be optimized independently and simplifies the overall implementation by avoiding the need to handle all functions in a monolithic structure.
Solution Approach 2:
The patent replaces the mechanical approach of comparing encrypted data blocks directly with a computational hashing approach. Instead of performing complex cryptographic comparisons or maintaining intricate key synchronization mechanisms, the system uses straightforward hash computation on plaintext data, which is computationally simpler and more scalable.
Data Source
AI summary
Deduplication of encrypted data using multiple keys includes responding to a request to store a predetermined set of data in an electronic data store by receiving a hash corresponding to the predetermined set of data, receiving encrypted data generated by encrypting the predetermined set of data using an encryption key, and receiving a key index corresponding to the encryption key. The hash may be determined to match a previously stored hash, the previously stored hash indicating that a previously encrypted version of the predetermined set of data is stored at a physical location in the electronic data store. Based on determining that the hash matches a previously stored hash, the hash, encrypted data, and key index are discarded.