Multi-thread File I/O System for Parallel Large Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file transfer technologies over the Internet face inefficiencies when transferring large files, particularly due to delays and loss of data fragments during multi-part transfers, which can lead to processing inefficiencies and the need for retries.
Innovation Solution
A multi-thread file input/output system that divides files into multiple pieces and transfers them in parallel using separate threads for transmission and reception, ensuring efficient processing and reassembly even when data fragments are received out of order or lost, by using a multi-thread structure for file input/output operations between computers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a file is divided into multiple pieces and transferred separately, then the transfer can be performed in parallel improving throughput, but delays and data loss occur more frequently requiring retries
Solution Approach 1:
The file is divided into multiple pieces (segments) that can be transferred in parallel through multiple threads, improving throughput while maintaining the ability to handle individual piece failures independently
Solution Approach 2:
The system implements acknowledgment mechanisms where the receiving end sends feedback to the transmitting end about successful receipt of data pieces, enabling automatic retry of lost pieces without affecting other pieces
2Productivity
If multiple threads are used for parallel file transfer, then processing efficiency improves, but system complexity increases
Solution Approach 1:
The file transfer process is segmented into independent threads, each handling specific data pieces, allowing parallel processing while maintaining manageable thread-level complexity
Solution Approach 2:
The multi-thread file I/O module provides universal functionality for both reading and writing operations across multiple threads, consolidating complexity into a single reusable component
3Productivity
If data fragments are received out of order, then parallel transfer efficiency is maintained, but correct reassembly becomes difficult
Solution Approach 1:
Sequence numbers are assigned to data pieces before transmission, enabling the receiving end to reorder pieces correctly upon arrival without affecting parallel transfer efficiency
Solution Approach 2:
The system uses acknowledgment feedback to track which pieces have been received and in what order, ensuring correct reassembly even when pieces arrive out of sequence
Data Source
AI summary
A configuration performing processing of dividing a file into a plurality of pieces and transmitting the same even when a size of the file is large in transfer of files (input/output) between computers on a network is provided. A multi-thread file input/output system includes a first module performing processing of reading data from an input file, dividing the data into a plurality of pieces, and transmitting the plurality of pieces to a network by multi-thread processing in a transmitter computer; and a second module performing processing of receiving the plurality of pieces from the network and integrating and writing the same to an output file 5 in a receiver computer.