Universal Connector Batching Small Files for NAS Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing devices face interruptions during application upgrades due to inefficient data backup processes, particularly in managing large sets of files and incremental backups in NAS systems, which affect system operation and resource utilization.
Innovation Solution
A universal connector is introduced that provides multi-threaded, protocol-agnostic data storage management services, optimizing backups by batching small files, managing incremental metadata requests, reducing network roundtrips, and utilizing resource availability to parallelize file processing and caching, thereby enhancing network bandwidth management and reducing data generation for incremental backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional backup methods are used for large sets of small files, then backup completeness is improved, but backup time and system interruption duration increase significantly
Solution Approach 1:
The backup system segments the large set of small files into batches, processing them in groups rather than individually. This segmentation allows the system to manage file operations more efficiently, reducing the total backup time while ensuring all files are captured in the backup set.
Solution Approach 2:
Multiple metadata requests and file operations are merged into single batched requests. By combining multiple small operations into larger batched operations, the system reduces the total number of network roundtrips and processing overhead, significantly decreasing backup time while maintaining complete file capture.
2Quantity of substance
If incremental backups are performed frequently to reduce data storage requirements, then storage efficiency is improved, but network bandwidth consumption and metadata generation increase
Solution Approach 1:
The system extracts and processes only the metadata necessary for incremental backup determination, separating this from full file data transfers. By taking out only the essential metadata for change detection and using batched processing, the system reduces network bandwidth consumption while maintaining storage efficiency through selective incremental backups.
3Measurement precision
If multiple metadata requests are sent to NAS system to track file changes, then backup accuracy is improved, but network roundtrips and processing overhead increase
Solution Approach 1:
Multiple metadata requests are merged into single batched requests that retrieve metadata for multiple files in one network roundtrip. This merging maintains backup accuracy by capturing all necessary file change information while significantly reducing network roundtrips and processing overhead compared to individual requests.
4Stability of the object's composition
If backup operations are performed sequentially to ensure data integrity, then data consistency is improved, but backup speed and resource utilization decrease
Solution Approach 1:
The backup operation is segmented into independent batched units that can be processed in parallel. Each batch maintains internal data consistency through proper transaction management, while multiple batches execute concurrently, significantly improving backup speed and resource utilization without sacrificing overall data integrity.
Solution Approach 2:
The system dynamically adjusts the parallelization level and batch processing based on available resources and data characteristics. This dynamic approach allows the system to maximize backup speed through parallel processing when resources permit, while maintaining data consistency through coordinated batch management.
Data Source
AI summary
A method for managing data includes obtaining, by a universal connector, a data share request to obtain data from a set of files, in response to the data share request: performing a share scan to obtain attributes associated with each file in the set of files, performing a batching of the set of files to obtain a set of file batches using the attributes and using a batch size threshold, performing a request conversion to generate a set of converted batch requests each corresponding to one of the set of file batches, and initiating a data share of the large set of files by sending the set of converted batch requests to a network attached storage (NAS) server.