Merkle Tree Data Integrity Verification for Outsourced Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in providing secure, efficient, and verifiable methods for data storage and retrieval, especially when data is stored by a second party that may not be trusted.
Innovation Solution
The use of a Merkle tree to ensure the integrity of data stored by a second party, where the data controller organizes the data into segments, hashes them in pairs, and retains a set of sample segments to verify the data's integrity and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is outsourced to a storage provider, then storage burden on data controller is reduced, but trust and verification of data integrity becomes problematic
Solution Approach 1:
The data is divided into multiple segments, and a Merkle tree is constructed from these segments. The data controller only needs to retain a small number of sample segments and the Merkle root, rather than the entire dataset. This segmentation allows efficient verification of data integrity while minimizing storage requirements at the controller side.
Solution Approach 2:
A Merkle tree serves as an intermediary mathematical structure between the data controller and storage provider. The Merkle root and sample segments act as verifiable proofs that enable the controller to confirm data integrity without directly accessing or storing the complete dataset, thus mediating the trust relationship.
2Reliability
If complete data is retained by data controller for verification, then data integrity can be ensured, but storage resources are consumed
Solution Approach 1:
Instead of retaining the complete dataset for verification, the system extracts only the essential components needed for integrity checking: a small number of sample segments and the Merkle root. These extracted elements are sufficient to verify the integrity of the entire outsourced dataset, dramatically reducing storage requirements.
Solution Approach 2:
The verification function is segmented into minimal representative samples rather than requiring the complete dataset. By selecting only critical sample segments and using the Merkle tree structure, the system achieves comprehensive verification coverage with minimal storage overhead.
3Reliability
If computational proofs are made complex to ensure security, then verification reliability improves, but processing time and resources increase
Solution Approach 1:
Instead of performing computationally intensive verification on the entire dataset, the system performs partial verification on a small number of sample segments. The Merkle tree structure allows the verification of these samples to provide strong cryptographic guarantees about the integrity of the complete dataset, achieving sufficient reliability with reduced computational effort.
Solution Approach 2:
The Merkle tree is pre-computed and the Merkle root is stored beforehand. During verification, the storage provider only needs to compute hashes of the provided segments and compare them against the pre-stored Merkle root, rather than performing complex verification computations on the entire dataset each time.
Data Source
AI summary
This application provides methods and systems for verifying safe, consistent and secure storage of data especially, but not limited to, situations where storage of the data is delegated to a third party. A data controller, Alice, takes at least one sample of her data D, performs an operation on it to produce a variation. She then calculates the root value of the Merkle tree that represents the data comprising the varied data sample. She sends her data to a storage provider, Bob, while retaining her sample(s) and the resulting Merkle root value(s). Alice does not tell Bob which sample(s) she has chosen, or the operations she has used in the variations, or any inputs to the operations. Alice can delete her original copy of the data. At a later date, Alice can verify that Bob still has her complete data and in its original state by requiring him to perform the same operation on the same data sample, calculate the root value of the resulting Merkle tree and send it to her. If Bob's root value matches Alice's root value, then Bob must have an original and complete copy of Alice's data otherwise he would not be able to calculate the correct Merkle root value. Embodiments can be arranged to fully automate the process, including implementing on a blockchain.