Shard-Level Synchronization for Cloud File Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data storage systems in cloud-based networks face significant processing overhead and resource utilization due to the need to create new versions of the entire file system upon changes, even if only a granular portion has been modified, leading to inefficiencies in maintaining directory structures, especially in large systems with many users.

Innovation Solution

Implementing a shared versioned file system that synchronizes data at the shard or sub-directory level, using filer servers to generate and export structured data representations in XML format, allowing for efficient propagation of changes only to the closest directory or sub-directory level, thereby reducing unnecessary processing and resource usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the entire file system is versioned upon any change, then data integrity and version control are maintained, but processing overhead and resource consumption increase significantly

Engineering Contradiction:

Improvedata integrityVSAvoidprocessing overhead

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The file system is divided into independent shards, where each shard can be versioned and synchronized independently. When a file changes, only the containing shard is updated rather than the entire file system, reducing processing overhead while maintaining data integrity through targeted version control at the shard level.

Inventive Principle:

Principle #1Segmentation

2Loss of information

If new versions of parent directories are created for every file change, then complete file system state is preserved, but processing time and resource usage increase

Engineering Contradiction:

Improvefile system state preservationVSAvoidprocessing time

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The versioning and synchronization operations are extracted from the global file system level and applied only to the specific shard level where changes occur. This eliminates the need to create and manage versions of entire parent directory structures, preserving necessary information at the shard level while significantly reducing processing time.

Inventive Principle:

Principle #2Taking out (Extraction)

3Difficulty of detecting and measuring

If the entire directory structure is walked to detect changes, then complete change tracking is achieved, but processing overhead increases significantly

Engineering Contradiction:

Improvechange detection accuracyVSAvoidprocessing overhead

Core Design Contradiction:

Difficulty of detecting and measuringVSProductivity

Solution Approach 1:

Change detection and tracking are applied locally at the shard level rather than globally across the entire directory structure. Each shard maintains its own change state, allowing for accurate change detection within the modified shard without the need to traverse and examine unrelated directories, thereby reducing processing overhead while maintaining detection accuracy.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11640374B2Shard-level synchronization of cloud-based data store and local file systems

Publication Date: 2023.05.02 NASUNI CORP

AI summary

An operations server synchronizes updates to a cloud-based shared versioned file system. The shared versioned file system includes directories and sub-directories that are divided into shards. The operations server coordinates requests from local filer servers, each running a respective local version of the shared versioned file system, to update a shard in the cloud-based shared versioned file system. The operations server can provide a global lock on the shard to a local filer server before it updates the shard in the cloud-based shared versioned file system.