Encrypted Data Deduplication Using a Distributed Ledger
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques are inefficient for encrypted data storage systems where there is no mutual trust between hosts, leading to unnecessary storage occupation and increased costs due to duplication of encrypted data.
Innovation Solution
A method utilizing a distributed ledger to manage and verify ownership of encrypted data blocks, preventing duplication by comparing hash values and updating ownership records, allowing deduplication even among untrusted hosts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data deduplication techniques are used on encrypted data, then data reduction is achieved, but trust between primary and backup systems is required which increases system complexity and security risks
Solution Approach 1:
A distributed ledger acts as an intermediary between primary and backup systems, enabling deduplication of encrypted data without requiring direct trust relationships. The ledger stores hash values and ownership indications, allowing verification of data uniqueness and ownership without exposing encrypted data or requiring mutual trust between systems.
Solution Approach 2:
The patent replaces the mechanical trust-based verification system with a cryptographic hash-based verification system recorded on a distributed ledger. Instead of relying on trust relationships to verify data uniqueness, the system uses hash value comparison and blockchain consensus mechanisms, eliminating the need for trust management while maintaining security.
2Reliability
If encrypted data is stored without deduplication, then security is maintained, but storage space is wasted and costs increase
Solution Approach 1:
The patent extracts the deduplication verification process from the encrypted data itself by storing only hash values and ownership indications on the distributed ledger. This allows the backup system to identify and eliminate duplicate encrypted data blocks without decrypting or exposing the actual data, maintaining security while improving storage efficiency.
Solution Approach 2:
Instead of storing duplicate encrypted data blocks, the system creates references (copies of hash values and ownership records) on the distributed ledger that point to the original data blocks. When deduplication is needed, the system references existing blocks rather than storing new copies, reducing storage space while maintaining data accessibility.
3Productivity
If hash values are compared directly between systems, then duplicate detection is efficient, but data privacy is compromised in untrusted environments
Solution Approach 1:
The distributed ledger serves as an intermediary that stores hash values and ownership indications in a decentralized, immutable manner. Systems can compare hash values recorded on the ledger without directly sharing or exposing encrypted data, maintaining privacy while enabling efficient duplicate detection through hash value matching and verification.
Data Source
AI summary
A method of preventing duplication of encrypted data in storage arranged in network in which two or more hosts can store data. The storage includes one or more encrypted data blocks, and the network further includes a distributed ledger accessible to each of the two or more hosts. The method leverages the distributed ledger for efficient deduplication of the data even with untrusted parties, such as the hosts. The method, performed by first host, includes obtaining a first hash value, a proof of ownership, and a random key for each block in plurality of new blocks of data. The method further includes generating and storing backup set description record, storing indication of the backup set description record, and determining zero or more blocks in plurality of new blocks that are present in storage to prevent duplication.