Dual Merkle Tree File System Backup Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in updating backup copies of file systems, particularly in cloud-based object storage systems, due to high network traffic and unnecessary read/write operations, which can be costly and resource-intensive.
Innovation Solution
The implementation of a dual Merkle tree structure, comprising a metadata Merkle tree and a data Merkle tree, allows for efficient updates of backup copies by calculating temporary Merkle trees and comparing keys to identify changed sub-trees, reducing the need for extensive read operations and minimizing network traffic through incremental updates and de-duplication techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional backup methods are used to synchronize file systems to cloud storage, then data redundancy is achieved, but network traffic and read/write operations increase significantly
Solution Approach 1:
The file system is divided into two separate Merkle trees: a metadata Merkle tree for directory structures and file metadata, and a data Merkle tree for actual file contents. This segmentation allows independent verification and updating of metadata and data, enabling selective synchronization that reduces unnecessary network traffic while maintaining data redundancy.
Solution Approach 2:
Instead of performing full backup operations, the system performs partial actions by comparing Merkle tree roots and updating only the specific sub-trees or nodes that have changed. This partial action approach maintains data redundancy while significantly reducing network traffic and write operations compared to complete resynchronization.
2Reliability
If full file system copies are synchronized to cloud storage, then data security is improved, but storage costs and write operations increase
Solution Approach 1:
The system creates cryptographic copies (Merkle trees) of the file system structure and data rather than physical copies. These Merkle trees serve as verified representations that enable security verification without requiring complete duplicate storage of all file contents, thereby reducing storage capacity requirements while maintaining data security through cryptographic verification.
Solution Approach 2:
The system changes the representation parameters of stored data from raw file contents to cryptographic hash-based Merkle tree structures. This parameter transformation enables efficient verification and selective updates, reducing the quantity of data that needs to be stored and transferred while maintaining security through the mathematical properties of hash functions.
3Productivity
If incremental updates are performed without verification, then update speed increases, but data integrity verification becomes complex
Solution Approach 1:
Merkle trees serve as intermediary structures between the file system and cloud storage verification processes. The tree structure provides intermediate hash values at each node, enabling efficient verification of incremental updates without requiring complex verification logic. Any change in the file system propagates up the Merkle tree, making it easy to identify and verify only the affected portions.
Solution Approach 2:
The system performs preliminary calculation of Merkle tree roots and intermediate node hashes before synchronization operations. This preliminary action prepares verification data in advance, enabling fast incremental updates with simple verification processes, as the cryptographic fingerprints of data blocks are already computed and stored in the Merkle tree structure.
Data Source
AI summary
A data storage apparatus includes an interface for communicating with a backup storage system, and one or more processors. The one or more processors are configured to store a local copy of at least part of a file system represented by hierarchical structure of directories and files, to store a backup copy of the at least part of the file system in the backup storage system, to store a dual Merkle tree that is calculated over the backup copy and includes (i) a metadata Merkle tree including Metadata Merkle-Tree Keys (MD-MTKs) and (ii) a data Merkle tree including Data Merkle-Tree Keys (D-MTKs), and to update the backup copy to match the local copy using the dual Merkle tree.