Backup Filesystem Walk Buffering for Sequential Disk Access

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In high-density filesystems, the traditional filesystem walk process for backup operations can consume up to 80% of the backup time, and while improving this with asynchronous backup processes reduces random disk reads, it still results in excessive disk activity due to multiple processes accessing nearby files randomly.

Innovation Solution

Implementing a system that queues modified objects into a buffer during the filesystem walk, sending the buffer for backup when full or crossing volume boundaries, ensuring sequential disk access and minimizing disk movement by reading nearby objects together.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional sequential filesystem walk is used for backup, then backup completeness is ensured, but backup time becomes excessively long (up to 80% of total backup time spent on filesystem walk)

Engineering Contradiction:

Improvebackup speedVSAvoidfilesystem walk time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The backup system divides the filesystem walk into multiple parallel threads, each handling a portion of the filesystem. This segmentation allows simultaneous discovery of modified objects across different directories, reducing the total time spent on filesystem traversal while maintaining complete backup coverage.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary identification of modified objects through asynchronous filesystem monitoring before actual backup operations begin. By pre-discovering which files need backup and organizing them into queues, the system eliminates the need for time-consuming sequential filesystem walks during the actual backup process.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If multiple asynchronous backup processes are used to improve parallelism, then backup throughput increases, but random disk reads increase significantly causing excessive disk activity

Engineering Contradiction:

Improvebackup throughputVSAvoidrandom disk reads

Core Design Contradiction:

ProductivityVSObject-generated harmful factors

Solution Approach 1:

The system merges the read operations of multiple backup processes by implementing a coordinated access scheme where multiple threads read from the same disk regions simultaneously. By combining read operations and using shared buffers, the system maintains parallel backup throughput while eliminating redundant random disk reads that would otherwise occur with independent backup processes.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

An intermediary buffer layer is introduced between the disk and backup processes. This buffer caches file data and serves multiple backup threads, allowing them to access data without direct repeated disk reads. The intermediary absorbs the random access patterns and presents sequential access to the disk, reducing disk activity while maintaining backup parallelism.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11726884B2Optimized filesystem walk for backup operations

Publication Date: 2023.08.15 EMC IP HLDG CO LLC

AI summary

The system for backing up data comprises an input interface configured to receive an input set of files. The system for backing up data additionally comprises a processor configured to add modified objects to a list of modified objects in the set of files, and, in the event it is determined to provide the list of modified objects to a backup process, provide the list to the backup process.