Universal Connector Parallel Backup for NAS File Batching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for managing backups in network attached storage (NAS) face inefficiencies in handling large numbers of files, frequent backups, incremental backups, and multiple protocols, leading to suboptimal performance and resource utilization.
Innovation Solution
The implementation of a universal connector that provides multi-threaded, protocol-agnostic data storage management, enabling batching of small files, incremental metadata processing, feedback-based batching based on resource availability, parallel processing of large folders, intelligent caching, and block-based incremental backups to optimize network bandwidth and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a backup system processes files sequentially in NAS, then the system complexity remains low, but the backup time and productivity deteriorate significantly when handling large numbers of files
Solution Approach 1:
The backup system segments the file system into multiple volumes or partitions and processes them in parallel using multiple threads. Each thread handles a specific segment independently, enabling concurrent backup operations that significantly reduce total backup time while managing complexity through structured division of work.
Solution Approach 2:
The system transitions from sequential single-threaded processing to multi-threaded parallel processing, adding the dimension of concurrency. This allows multiple backup operations to execute simultaneously, improving productivity from a linear time complexity to a parallel time complexity model.
2Reliability
If the backup system performs frequent full backups, then data reliability is improved, but resource consumption and data generation increase
Solution Approach 1:
The system performs preliminary identification of changed files by comparing metadata (timestamps, sizes, checksums) before initiating backup. This preliminary action filters out unchanged files, allowing the backup system to focus resources only on actual changes rather than transferring all data repeatedly.
Solution Approach 2:
The system extracts only the necessary changed files and blocks for backup instead of transferring entire folders or all files. By identifying and isolating only the modified portions, the system reduces data generation and network bandwidth consumption while maintaining reliable backup of actual changes.
3Productivity
If the backup system processes small files individually, then processing precision is maintained, but the number of roundtrips and time consumption increase
Solution Approach 1:
The system merges multiple small file operations into batch requests. Instead of processing files one at a time with separate system calls, the backup system aggregates file operations into single requests that can be handled in parallel, reducing the total number of roundtrips to the NAS while maintaining accurate processing of each file.
4Adaptability or versatility
If the backup system uses protocol-specific handling, then adaptability to different protocols is maintained, but device complexity and ease of operation deteriorate
Solution Approach 1:
The backup system implements a universal connector that provides a common interface for multiple storage protocols (SMB, NFS, AFP, etc.). This universal layer abstracts protocol differences, allowing the same backup logic to work across different protocols without requiring separate specialized handlers for each, thereby reducing implementation complexity while maintaining broad adaptability.
Data Source
AI summary
A method for managing data include obtaining, by a universal connector, a backup request for a set of files, in response to the backup request: performing a batching of file read requests for data associated with the set of files to obtain a set of file read batches, sending the set of file read batches to a network attached storage (NAS) server, obtaining a response for each of the set of file read batches from the NAS server, wherein the response for each of the set of file read batches comprise a portion of the data, wherein at least two responses for the set of file read batches are obtained in parallel, writing the portion of the data to a backup container in parallel with the obtaining, and initiating a backup of the set of files using the backup container after the writing is completed.