Intelligent File System Slicing for NAS Backup Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data backup systems for network attached storage (NAS) face challenges such as proprietary snapshot limitations, limited scalability of NDMP, and slow file system crawlers, which hinder efficient data recovery and processing times.
Innovation Solution
Implementing an intelligent file system slicer that breaks down the file system into slices, allowing backup agents to process data in parallel, using techniques like depth-based, size-based, and file count-based slicing to maximize throughput and reduce processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If file system crawlers are used to traverse files over network connection, then data backup can be performed, but processing time becomes excessively slow
Solution Approach 1:
The patent divides the file system into multiple slices based on depth, size, or file count metrics. Each slice is independently processed by parallel backup agents, eliminating the sequential traversal bottleneck of traditional crawlers and enabling concurrent data backup operations across multiple file system portions simultaneously
Solution Approach 2:
The patent introduces slicing dimensions (depth-based, size-based, file count-based) to organize and process the file system structure. By adding these organizational dimensions, the system can systematically divide and conquer the file system traversal task, enabling parallel processing without losing track of file locations and relationships
2Reliability
If array snapshots are used for data protection, then point-in-time copies can be created, but scalability is limited to same device/model restoration
Solution Approach 1:
The patent creates a universal backup system that can restore data to multiple target devices and models, not just the original source device. The backup agents and slice management structure enable cross-platform, cross-vendor restoration capabilities, making the backup system versatile and adaptable to different hardware configurations
3Speed
If NDMP protocol is used for data transport, then backup speed is enhanced, but parallel capabilities are limited and scaling to hundreds of terabytes is difficult
Solution Approach 1:
The patent segments the backup workload into multiple independent slices that can be processed in parallel by multiple backup agents simultaneously. This segmentation approach enables the system to scale to handle hundreds of terabytes by adding more parallel processing agents, overcoming the single-protocol limitation of NDMP
Solution Approach 2:
The patent combines multiple slicing strategies (depth-based, size-based, file count-based) and merges their results into a coordinated parallel processing framework. This merging of approaches allows the system to leverage the speed benefits of protocols like NDMP while simultaneously achieving high parallel capability through multi-agent processing
4Productivity
If file system crawlers run in parallel with first pass layout determination, then backup throughput can be improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent performs preliminary slicing of the file system structure before the actual backup operation begins. By pre-organizing the file system into slices based on depth, size, or file count during an initial setup phase, the system eliminates the need for complex runtime layout determination, reducing processing overhead during parallel backup operations
Data Source
AI summary
Embodiments are described for storing array snapshots of a block-based system in networked storage, such as an NAS or SAN device. A system and process determines a size of the block device and splits it into a number of slices based on the size of the block device, with each slice comprising slice data. The slice data is written to protection storage either directly or through temporary buffer memory. The slice number and memory location for the slice data is stored in a key/value map, where the key comprises the slice number, and the value comprises the location. Backup agents are deployed to back up the slice data for each slice of the block device to the networked storage, and the slice data is stored as array snapshots in the network storage.