Trusted Server Data Deduplication in Untrusted Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems in untrusted environments, such as cloud storage, face inefficiencies in data deduplication and compression due to the need for mutual trust between servers and storage systems, leading to reduced data reduction and increased storage costs.
Innovation Solution
A method and system that employs a set of trusted servers to compress and encrypt data using a specific encryption key and hash function, allowing for deduplication and compression in untrusted storage systems without mutual trust, by using an API to manage key IDs and hash values, enabling efficient data storage and reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted in untrusted storage systems, then data security is improved, but data compression and deduplication capabilities deteriorate
Solution Approach 1:
The system performs compression and deduplication operations on data before encryption is applied. By preparing the data in advance (compressing and deduplicating) before the encryption step, the system achieves both security and data reduction without the limitations of encrypting first. This preliminary action resolves the contradiction by changing the sequence of operations.
Solution Approach 2:
The system segments data into fixed-size aligned blocks before processing. This segmentation enables efficient compression and deduplication operations on individual blocks, which are then encrypted separately. The block-based approach maintains security while enabling substantial data reduction through compression and deduplication of segmented data units.
2Productivity
If mutual trust is established between servers and storage systems, then compression and deduplication efficiency is improved, but system complexity and security risks increase
Solution Approach 1:
The system introduces a trusted server as an intermediary between the untrusted storage system and the data owner. The trusted server performs compression and deduplication operations, acting as a mediator that enables efficient data processing without requiring the storage system itself to be trusted. This intermediary approach maintains efficiency while reducing system complexity and security risks associated with trusting the storage system directly.
Solution Approach 2:
The system enables the trusted server to autonomously perform compression and deduplication operations without requiring continuous verification or coordination with the storage system. The trusted server independently processes data blocks, manages encryption keys, and performs deduplication checks, providing self-service functionality that improves efficiency while maintaining security without adding complex inter-system coordination mechanisms.
3Reliability
If encryption keys are frequently changed, then data security is improved, but deduplication capability deteriorates
Solution Approach 1:
The system performs deduplication operations before encryption with potentially changing keys. By identifying and eliminating duplicate data blocks before the encryption step, the system ensures that deduplication effectiveness is not compromised by subsequent key changes. The deduplication is established in advance, and then encryption with any key maintains this deduplication structure.
Solution Approach 2:
The system maintains copies of data block hashes and metadata that are independent of encryption keys. These copies enable deduplication identification to persist across key changes, as the deduplication information is stored separately from the encrypted data. This copying approach allows the system to maintain high deduplication ratios even when encryption keys are frequently rotated for security purposes.
Data Source
AI summary
A method for data deduplication and compression in untrusted storage system is provided for storing large amount of data more efficiently and in a secure manner and by maintaining the integrity of the data. Such data deduplication and compression in untrusted storage system is achieved by utilizing by a system comprising a set of trusted servers, which are configured to trust each other and to share common encryption keys.