Distributed Metadata Store for Backup Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data stores face challenges in maintaining consistency while ensuring high availability and partition tolerance, leading to reduced consistency in read/write access, which is critical as companies rely heavily on data stored on their networks.
Innovation Solution
The system employs a method of splitting metadata from file data, storing metadata separately, and using manifest files for local change detection to improve backup and replication performance, with asynchronous replication correctness validation and network bandwidth optimization to ensure efficient data transfer and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high-speed read/write access is enabled in distributed data stores, then availability is improved, but consistency deteriorates
Solution Approach 1:
The patent segments data into two distinct components: metadata stored in a distributed data store for fast access, and file data stored in traditional filesystems for consistency. This segmentation allows each component to optimize for its specific requirements - the metadata store provides high availability and speed, while the filesystem maintains consistency for actual data storage.
Solution Approach 2:
The patent extracts metadata from the traditional filesystem and places it in a separate distributed data store. By taking out metadata management from the filesystem, the system can leverage the speed and availability of distributed stores without compromising filesystem consistency, as only the lightweight metadata is moved rather than the entire file data.
2Productivity
If metadata is stored separately from file data, then backup performance is improved, but device complexity increases
Solution Approach 1:
The patent implements a universal mapping layer that works across different filesystem types (NTFS, ext3, HFS+, FAT32). This mapping layer provides multi-functionality by abstracting the complexity of different filesystem formats, allowing the system to backup and replicate data from various sources without requiring filesystem-specific handling, thus improving performance while managing complexity through standardization.
Solution Approach 2:
The patent introduces a mapping layer as an intermediary between the distributed data store and traditional filesystems. This mediator handles the complexity of translating between different storage formats and protocols, simplifying the overall system architecture by providing a uniform interface for metadata storage while maintaining compatibility with diverse filesystems.
3Ease of operation
If traditional backup systems use central server catalogs, then ease of operation is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent performs preliminary action by storing metadata locally on the client system before backup operations. This local metadata cache allows the client to quickly identify which files need backing up without contacting the central server for every decision, significantly reducing network bandwidth consumption while maintaining ease of operation through automated local change detection.
Solution Approach 2:
The patent enables self-service by allowing the client system to autonomously determine what needs to be backed up using locally stored metadata and change detection mechanisms. The client independently makes backup decisions based on its own state, eliminating the need for continuous central server guidance and reducing network communication overhead while maintaining operational simplicity.
Data Source
AI summary
Systems and methods are disclosed for backing up a computer with a distributed meta data store by splitting meta data from file data; storing the meta data for each file in a separate file; mapping the file data and meta data to an arbitrary file system; and reconstructing files for the arbitrary file system.