Cloud Backup Agent With Concurrent Proxy Uploads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems are inefficient and time-consuming due to over/underutilization of proxy servers and repeat scans caused by interruptions, especially in cloud-based storage solutions.
Innovation Solution
A backup agent utilizing a master node and multiple proxy nodes to concurrently scan and upload batches of data, with checkpointing operations to minimize repeat scans and ensure efficient data transfer to the cloud network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple proxy servers are used to scan and upload large scanned data sections, then the backup operation can handle large data volumes, but it causes over or underutilization of proxy servers
Solution Approach 1:
The patent implements dynamic proxy server selection and utilization. The system dynamically determines which proxy servers should be used based on real-time conditions, data sizes, and server availability. This dynamic approach allows the system to optimize proxy server utilization by assigning tasks adaptively rather than using a fixed number of servers, thereby avoiding both overutilization and underutilization while handling large data volumes efficiently
2Productivity
If regular multithreaded scanning is used with multiple threads scanning independent directories, then the scanning operation can process data in parallel, but interruptions cause the entire data to be scanned again upon resumption
Solution Approach 1:
The patent implements preliminary actions by creating a comprehensive index of all data sections before the actual scanning and backup process begins. This index structure pre-organizes data information about directories, files, and their relationships, allowing the system to quickly locate and resume scanning from the exact interruption point without needing to re-scan entire data structures. This preliminary indexing action eliminates the need for repeat scans upon interruption while maintaining parallel processing capabilities
Solution Approach 2:
The system incorporates feedback mechanisms that continuously monitor scanning progress and track which data sections have been processed. When an interruption occurs, the feedback information about completed and ongoing scanning operations enables the system to resume from the precise location without repeating already scanned sections. This feedback loop ensures efficient resource utilization and time management during interrupted operations
3Device complexity
If a single proxy server is used to scan and upload data, then the system structure is simple, but the backup operation becomes slow and inefficient for large data sets
Solution Approach 1:
The patent applies segmentation by dividing the backup operation into multiple independent sections or tasks that can be handled by different proxy servers simultaneously. Instead of using a single proxy server for the entire data set, the system segments the data into manageable portions and distributes them across multiple proxy servers. This segmentation enables parallel processing while maintaining relatively simple individual server structures, thereby increasing overall backup speed without significantly complicating the system architecture
Data Source
AI summary
A backup agent for performing a backup operation is provided. The backup agent comprises a memory storing one or more processor-executable routines and a processor communicatively coupled to a data storage system and configured to access unstructured data stored in therein. The processor comprises a master node and a plurality of proxy nodes. The master node is configured to perform a backup operation by generating a plurality of threads to perform a scan operation; wherein during the scan operation a plurality of batches of data stored on a data storage device is scanned. The processor is further configured to create a global queue of the plurality of batches of data upon completion of the scan and assign the plurality of batches of data from the global queue to a plurality of proxy nodes and/or to the master node. The master node and each proxy node are configured to perform an upload operation by uploading the assigned batch of data on the cloud network; wherein the master node and the proxy nodes operate concurrently.