Distributed Snapshot Scaling via Dynamic File Sharding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed data management and storage systems face challenges in scaling snapshotting performance for large files, as they often underutilize additional nodes when dealing with single large files, leading to inefficiencies in data distribution and processing.

Innovation Solution

The system dynamically adjusts file partitioning or sharding, distributing large files across multiple nodes based on file size, available disk space, and data transmission rates, allowing for parallel fetching and storage of file shards across the cluster, thereby scaling snapshotting performance with the number of nodes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the system stores large files as single files, then file integrity is maintained, but snapshotting performance cannot scale across multiple nodes

Engineering Contradiction:

Improvesnapshotting performanceVSAvoidfile partitioning complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system divides large files into smaller shards distributed across multiple nodes in the cluster. Each node stores a portion of the file, enabling parallel snapshot operations. The segmentation is dynamic, adjusting shard size based on file size and cluster configuration, allowing snapshotting performance to scale with the number of nodes while maintaining file integrity through coordinated reconstruction.

Inventive Principle:

Principle #1Segmentation

2Productivity

If the system partitions large files into shards, then snapshotting performance scales across nodes, but data distribution efficiency decreases

Engineering Contradiction:

Improvesnapshotting performanceVSAvoiddata distribution time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-calculating optimal shard sizes and distribution strategies before snapshotting begins. File metadata is analyzed in advance to determine partitioning schemes, and nodes are pre-configured with their assigned shards. This preliminary preparation minimizes data distribution time during actual snapshot operations while maintaining scalable performance across the cluster.

Inventive Principle:

Principle #10Preliminary action

3Quantity of substance

If additional nodes are added to the cluster, then storage capacity increases, but snapshotting performance remains underutilized for single large files

Engineering Contradiction:

Improvestorage capacityVSAvoidsnapshotting performance

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The system implements dynamic file partitioning that automatically adapts to the number of available nodes in the cluster. When nodes are added, the system dynamically recalculates shard distribution to utilize all available resources. This dynamic adjustment ensures that snapshotting performance scales linearly with cluster size, converting additional storage capacity into proportional performance gains through optimized parallel processing across all nodes.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12130711B2Scaling single file snapshot performance across clustered system

Publication Date: 2024.10.29 RUBRIK INC

AI summary

In some embodiments, a process for restoring a version of a virtual machine using a data storage system comprises identifying a particular version of a virtual machine to be restored, determining a base image from which the particular version may be derived, determining a set of incremental files for generating the particular version, generating a file associated with the particular version using the base image and the set of incremental files, and outputting at least a portion of the file.