Parallel Backup Thread Workload Balancing via Directory Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing backup processes require computationally expensive methods to divide data sets into equally sized saveset groups for parallel processing, leading to inefficient workload distribution across multiple threads.

Innovation Solution

Utilizing known characteristics of directory structures, such as files being stored in a four-level hierarchy, to generate a worklist of lowest level directories and divide them into savesets of similar size without traversing the entire directory structure, enabling efficient and balanced workload distribution across multiple threads.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If traditional exhaustive matching methods are used to divide data into saveset groups, then equal sized groups are achieved, but computational cost and time consumption increase significantly

Engineering Contradiction:

Improveequality of saveset group sizesVSAvoidtime to traverse directory structure

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent segments the directory structure traversal task by utilizing known characteristics (such as four-level hierarchy) to directly generate worklists of lowest-level directories without exhaustive traversal. This segmentation allows rapid division of data into saveset groups while maintaining relatively equal sizes, resolving the contradiction between precision in group sizing and time consumption.

Inventive Principle:

Principle #1Segmentation

2Ease of operation

If exhaustive directory traversal is performed to create balanced saveset groups, then workload distribution is optimized, but processing speed decreases

Engineering Contradiction:

Improveworkload distribution balanceVSAvoidbackup processing speed

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent applies preliminary action by using known directory structure characteristics to pre-determine the organization of saveset groups before actual backup operations begin. By pre-generating worklists of lowest-level directories and organizing them into balanced savesets in advance, the system achieves both good workload distribution and high processing speed during the actual backup operation.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If computationally expensive exhaustive matching is used, then optimal saveset combinations are found, but system resources are consumed excessively

Engineering Contradiction:

Improveoptimality of saveset divisionVSAvoidcomputational energy consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent employs a lightweight, efficient algorithm that uses known directory characteristics to rapidly generate saveset divisions without requiring computationally expensive exhaustive matching. This approach achieves sufficiently optimal division results while consuming minimal computational resources, effectively replacing the expensive exhaustive matching process with a more efficient method.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS7574461B1Dividing data for multi-thread backup

Publication Date: 2009.08.11 DELL EMC

AI summary

Performing a backup is disclosed. One or more directories at a lowest level in a file directory associated with a storage is identified. The identified lowest level directories are divided into one or more directory groups. The backup of the directory groups is performed using parallel threads until all directory groups have been backed up.