Scalable File Replication via Blob Store Decoupling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-master file replication systems face challenges in scalability, management, and reliability, especially when dealing with a large number of subscribers, as they often require costly infrastructure and can experience synchronization issues between replication metadata and file system contents, leading to frequent rescans.
Innovation Solution
A scalable file replication system is implemented using a binary large object (blob) store that decouples file system metadata and blob storage, merging replication metadata with scalable store metadata for simultaneous transaction, eliminating the need for rescans and providing a 'drive in the sky' facility for automatic file replication and web-based access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multi-master file replication systems are used to synchronize files between computing devices, then file synchronization capability is improved, but system complexity and cost increase when scaling to large numbers of subscribers
Solution Approach 1:
The patent introduces a central server as an intermediary that coordinates file replication between multiple computing devices. Instead of direct peer-to-peer multi-master replication, the server acts as a mediator that receives file changes from any device and propagates them to other devices, simplifying the replication logic and reducing system complexity while maintaining synchronization reliability
Solution Approach 2:
The patent separates file metadata storage from actual file content storage by introducing a distributed file system namespace layer. This dimensional separation allows the replication system to manage file metadata centrally while storing actual file contents in distributed blob storage, reducing the complexity of synchronizing entire file systems while maintaining file accessibility
2Loss of information
If replication metadata is stored separately from file system contents, then replication tracking is improved, but synchronization issues occur leading to frequent rescans
Solution Approach 1:
The patent merges replication metadata with the distributed file system namespace metadata. Instead of maintaining separate replication metadata structures that can become out of sync, the replication state is integrated into the file system's own metadata layer, ensuring consistency between replication tracking and file system contents without requiring separate synchronization mechanisms or rescans
3Ease of operation
If full file system semantics are implemented in the replication system, then file access capability is improved, but scalability and management complexity worsen
Solution Approach 1:
The patent segments the file system into two independent layers: a distributed file system namespace layer that handles file metadata, permissions, and access semantics, and a blob storage layer that handles actual file contents. This segmentation allows the replication system to operate at the namespace layer with full file system semantics while the blob storage layer provides simple, scalable content storage, improving both file access capability and scalability
Data Source
AI summary
Various embodiments introduce the notion of a replication entity which implements a highly scalable file replication system. In one embodiment, the replication entity resides in the form of a scalable replication service. In at least some embodiments, the replication service provides a “drive in the sky” facility that can be used by individuals, such as subscribers, to synchronize their individual machines, such that their files are automatically replicated to a safe and always-on location. Alternatively or additionally, individuals such as subscribers can also access their files via a web-based interface when they are away from their machines.