Differential Deduplication for Encrypted Untrusted Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems fail to achieve effective deduplication and compression in untrusted storage environments due to the lack of mutual trust between servers and storage, leading to lower deduplication and compression ratios for encrypted data.
Innovation Solution
A method and system for differential deduplication in untrusted storage, involving a trusted server that compresses and encrypts data blocks, identifies similar blocks in the untrusted storage, and performs differential deduplication by using reference blocks for further compression and encryption, thereby enhancing deduplication and compression ratios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted in untrusted storage to ensure security, then data security is improved, but deduplication and compression ratios deteriorate because encrypted data cannot be effectively compressed or deduplicated
Solution Approach 1:
The data is divided into fixed-size aligned blocks before encryption. Each block is independently processed through compression and encryption, allowing deduplication at the block level rather than requiring full-file deduplication. This segmentation enables effective deduplication of encrypted data by comparing hash signatures of individual blocks.
Solution Approach 2:
Data compression is performed before encryption in the trusted server environment. By compressing plaintext data first and then encrypting the compressed result, the system achieves both security and space efficiency. The preliminary compression action reduces the size of data before it becomes encrypted and stored in untrusted storage.
2Quantity of substance
If conventional deduplication techniques are applied to encrypted data in untrusted storage, then storage space is reduced, but deduplication ratios deteriorate because encrypted data blocks cannot be effectively compared for similarity
Solution Approach 1:
Similarity hash values are calculated in advance on plaintext data blocks before encryption and storage. These pre-computed hash values are stored alongside the encrypted data blocks, enabling the untrusted storage to perform efficient deduplication by comparing hash values rather than decrypting and comparing actual data content, thus maintaining high deduplication ratios.
Solution Approach 2:
Instead of storing multiple copies of identical encrypted data blocks, the system stores only one copy and uses hash-based pointers or references to identify duplicate blocks. When a duplicate block is detected through hash comparison, the system creates a reference to the existing block rather than storing redundant data, achieving significant storage space reduction.
3Reliability
If data is stored without compression in untrusted storage to maintain security, then data security is improved, but storage efficiency deteriorates leading to higher costs and reduced performance
Solution Approach 1:
Data compression is performed as a preliminary step before encryption and storage in the untrusted environment. The trusted server compresses the plaintext data, then encrypts the compressed data. This preliminary compression action ensures that the data occupying space in untrusted storage is already minimized, achieving both security and storage efficiency without requiring decompression operations in the untrusted environment.
Data Source
AI summary
The present disclosure provides a method and system for differential deduplication of data blocks in an untrusted storage. An example system determines first data blocks of similar data blocks as reference data blocks and second data blocks of the identified similar data blocks which are compressed using the reference data blocks to enable differential deduplication of data blocks.