Dynamic Scanning Thread Adjustment for Network Backup Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Companies face challenges in efficiently backing up large volumes of data stored in network-based storage environments without the ability to install data agents or filter drivers, particularly in third-party managed systems, which limits their control over the underlying systems and resource utilization.
Innovation Solution
A data agent within the primary storage environment determines the optimal number of scanning threads to identify files for backup in a network-based storage environment, analyzing network performance and using historical analysis to dynamically adjust the number of threads, enabling efficient backup operations even in remote or third-party managed storage environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a data agent or filter driver is installed on a system in the primary storage environment, then the ability to monitor and control backup operations is improved, but the ease of operation deteriorates due to permission requirements and system access constraints
Solution Approach 1:
The patent introduces a filter driver as an intermediary component that operates at the file system level without requiring direct system access or administrative permissions. This filter driver monitors file operations and communicates with the data agent, enabling reliable backup control while avoiding the need for the data agent to directly access or modify system configurations.
2Productivity
If the number of scanning threads is increased to improve backup speed, then productivity is improved, but the use of energy and computational resources worsens
Solution Approach 1:
The patent implements dynamic thread pool management where the number of scanning threads is adjusted in real-time based on system conditions, file system activity, and available resources. The filter driver monitors system state and dynamically scales the thread pool size, allowing the system to maximize backup speed when resources are abundant while reducing computational overhead when system resources are constrained.
Solution Approach 2:
The system incorporates feedback mechanisms where the filter driver continuously monitors file operation patterns, system performance metrics, and resource availability. This feedback is used to adjust the scanning thread configuration, creating a closed-loop control system that optimizes the balance between backup productivity and resource consumption based on actual system conditions.
3Loss of time
If differential backups are performed to reduce resource requirements, then the loss of time is reduced, but the measurement precision deteriorates due to the need to monitor file creation and writes
Solution Approach 1:
The filter driver performs preliminary monitoring of file operations as they occur, maintaining a record of file creation, modification, and deletion events. This preliminary action allows the differential backup process to quickly identify changed files without performing exhaustive scans, reducing backup time while maintaining accurate detection of file changes through the pre-captured operation logs.
Data Source
AI summary
Embodiments disclosed herein address the need to more efficiently backup a network-based storage environment that may be remote from a primary storage environment. For example, embodiments herein can provide a more efficient backup of a storage managed by a third-party entity. To improve the backup process, embodiments herein may optimize the number of scanning threads that are used to identify files that are to be backed up by, for example, analyzing the characteristics of the network and/or the network storage system to determine a number of scanning threads that will enable faster scanning of the network storage system while at the same time not overburden or be slowed down by a network between the network-based storage environment and the primary storage environment.