Shard-Level Cloud File System Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face significant processing overhead and resource utilization due to the need to create new versions of an entire file system upon changes, even when only a granular portion of the system has been modified, leading to inefficient data synchronization in large file systems with many users.
Innovation Solution
Implementing a shared versioned file system that synchronizes data at the shard or sub-directory level, using XML-based structured data representations and filer servers to manage changes locally before updating cloud storage, thereby limiting propagation of changes to the closest directory or sub-directory level, reducing unnecessary processing and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire file system is versioned upon any change, then data integrity and version control are maintained, but processing overhead and resource consumption increase significantly
Solution Approach 1:
The file system is divided into independent shards (sub-directories), where each shard can be versioned and synchronized independently. This segmentation allows only the affected shard to be updated rather than the entire file system, reducing processing overhead while maintaining data integrity through selective versioning.
2Measurement precision
If the entire directory structure is walked to detect changes, then accurate change detection is achieved, but time consumption and computational resources increase
Solution Approach 1:
The directory structure is segmented into shards, and change detection is performed only on the specific shard where a change occurred rather than walking the entire directory structure. This maintains accurate change detection for the affected area while significantly reducing time consumption and computational resources.
3Reliability
If new versions of parent directories are created for every file change, then complete version history is preserved, but processing overhead and resource usage increase
Solution Approach 1:
Version history is maintained at the shard level rather than creating new versions of entire parent directory structures. Each shard maintains its own version history independently, preserving version completeness for modified portions while reducing processing overhead by avoiding unnecessary version creation in unaffected areas.
4Productivity
If granular portion synchronization is implemented, then processing overhead is reduced, but system complexity increases due to sharding management
Solution Approach 1:
The file system is segmented into shards managed by filer servers, enabling granular synchronization that improves processing efficiency. The complexity of sharding management is handled by the filer server infrastructure, which provides automated shard management and coordination.
Data Source
AI summary
An operations server synchronizes updates to a cloud-based shared versioned file system. The shared versioned file system includes directories and sub-directories that are divided into shards. The operations server coordinates requests from local filer servers, each running a respective local version of the shared versioned file system, to update a shard in the cloud-based shared versioned file system. The operations server can provide a global lock on the shard to a local filer server before it updates the shard in the cloud-based shared versioned file system. Preferably, shards are sized dynamically based on the number of entries in the directories associated therewith.