Encrypted Data Chunk De-duplication via Metadata Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage facilities face challenges in securely storing large data objects without duplicating them, especially when the objects are encrypted, as existing de-duplication methods are difficult to implement effectively in encrypted formats.
Innovation Solution
A storage service divides large binary objects into encrypted data chunks, assigns unique encryption keys and message authentication codes, and stores these chunks separately, along with metadata containing encryption keys and access control lists, to manage storage efficiently and securely, avoiding duplication by verifying integrity and decrypting data upon access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If encryption is applied to secure large data objects, then security is improved, but de-duplication becomes difficult
Solution Approach 1:
The patent segments large data objects into smaller chunks before encryption. Each chunk can be independently processed and compared for de-duplication. The system divides the original data object into multiple data chunks, encrypts them separately, and stores them as individual units with associated metadata, enabling granular de-duplication operations on encrypted data without requiring decryption.
2Quantity of substance
If de-duplication is implemented to reduce storage, then storage efficiency is improved, but it becomes difficult when objects are encrypted
Solution Approach 1:
The patent performs de-duplication checks on encrypted data chunks before final storage by comparing cryptographic hashes or metadata of the encrypted chunks against existing stored chunks. This preliminary comparison allows the system to identify and eliminate duplicate encrypted data without requiring decryption, thereby maintaining storage efficiency while working within the constraints of encrypted data.
3Quantity of substance
If large data objects are stored without duplication, then storage space is optimized, but verification of data integrity becomes more challenging
Solution Approach 1:
The patent implements feedback mechanisms through metadata storage that includes cryptographic hashes, authentication tags, or checksums for each encrypted data chunk. When data is retrieved or verified, the system recomputes these verification values and compares them against the stored metadata to confirm data integrity. This feedback loop enables accurate integrity verification of de-duplicated encrypted data without requiring storage of redundant copies.
Data Source
AI summary
A storage service receives a binary large object (blob) for storage, and the service creates first and second sets of data chunks from the blob. The chunks in the first set together equal the blob, and the service uses one or more encryption keys to encrypt each of the data chunks in the first set. The chunks in the second set also together equal the blob. The service assigns a message authentication code (MAC) to each data chunk in the second set. The service stores the encrypted data chunks in one or more data stores, and it stores the encryption keys and the MACs as metadata in a metadata memory.