Parallel File Transfer Agent with Queue-Based Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file transfer methods across networks are inefficient due to their reliance on low-level operations in network file systems, leading to slower data transfer speeds during server migrations, especially when dealing with large amounts of data.
Innovation Solution
The method involves identifying and performing separate operations in parallel, such as file list creation, reading, compressing, encrypting, sending, receiving, decrypting, unpacking, and writing, using queues and multiple network connections to optimize resource utilization, particularly CPU, disk, and network resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional file transfer utilities use network file systems, then file transfer can be performed across networks, but low-level operations take much more time resulting in slower data transfer
Solution Approach 1:
The patent segments the file transfer process into distinct operations (file listing, reading, compressing, encrypting, sending, receiving, decrypting, unpacking, writing) that can be executed in parallel. Each operation is performed through separate procedures working simultaneously on different data streams, eliminating the sequential bottleneck of conventional utilities.
Solution Approach 2:
The patent transitions from single-threaded sequential processing to multi-threaded parallel processing by introducing multiple concurrent procedures for each operation type. This dimensional shift from one-dimensional sequential execution to multi-dimensional parallel execution dramatically increases throughput while maintaining network file system compatibility.
2Device complexity
If rsync utility performs all operations in succession using a single network connection, then resource usage is simplified, but data transfer speed is reduced
Solution Approach 1:
The patent divides the monolithic rsync operation into multiple independent procedural segments (file listing, reading, compressing, encrypting, sending, receiving, decrypting, unpacking, writing) that can execute concurrently. Each segment processes different portions of the data stream simultaneously, transforming the single-connection sequential model into a multi-connection parallel model.
Solution Approach 2:
The patent ensures continuous utilization of network bandwidth and system resources by maintaining multiple active data transfer streams simultaneously. While one connection is transmitting data, others are establishing connections, compressing data, or receiving data, eliminating idle periods and maximizing resource utilization throughout the transfer process.
3Productivity
If multiple network connections are used for parallel operations, then data transfer speed increases, but resource management complexity increases
Solution Approach 1:
The patent introduces queue data structures as intermediary components between parallel procedures and network connections. These queues buffer data between production procedures (reading, compressing, encrypting) and consumption procedures (sending, receiving, decrypting), automatically managing the complexity of coordinating multiple concurrent operations and connections.
Solution Approach 2:
The parallel procedures automatically self-regulate their execution based on queue states and resource availability. When queues are full or connections are busy, procedures automatically wait or adjust their operation rate, eliminating the need for complex external coordination mechanisms while maintaining efficient resource utilization.
Data Source
AI summary
System for transferring files includes source and target servers. A client on target server initiates data transfer by sending requests to agent on source server, including local path to data. Number of requests equals number of connections. Agent receives requests, uploads data from local storage and transfers data to client. Client receives data and stores it locally. File transfer process has multiple file copy operations. Each operation is executed by multiple parallel and independent procedures. File data and metadata are transferred between procedures via limited size queues, in random order. Each procedure is blocked when adding data to queue when queue reaches maximum size, or when queue is empty and not marked closed. The file list generation procedure asynchronously launches copies of itself to generate list of files in parallel. Agent reads next file from the list only when all blocks of current file are distributed between the copy procedures.