Parallel Data Transfer via High-Speed and Integrity Connections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transfer protocols, such as TCP and UDP, face a trade-off between speed and reliability, with TCP ensuring high integrity but slow speeds and UDP providing fast transfers at the cost of data integrity, limiting their effectiveness in balancing performance and reliability for large file transfers.
Innovation Solution
A method involving a high-speed connection (e.g., UDP) and a high-integrity connection (e.g., TCP) is established to facilitate parallel data transfer, where a descriptor file tracks the status of data blocks, ensuring reliable delivery by resending missing blocks over the high-integrity connection, while the high-speed connection handles the bulk data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If TCP protocol is used for data transfer, then data integrity is improved, but data transfer speed deteriorates
Solution Approach 1:
The data transfer process is segmented into two independent pathways: a high-speed path for bulk data transmission and a separate reliability management path for tracking and retransmission. This segmentation allows the bulk transfer to proceed at maximum speed while reliability checks are handled separately, resolving the contradiction between speed and integrity.
Solution Approach 2:
A descriptor file acts as an intermediary between the high-speed data transfer and the reliability verification processes. The descriptor file tracks received data blocks and coordinates retransmissions without blocking the main data flow, enabling both high speed and high reliability simultaneously.
2Speed
If UDP protocol is used for data transfer, then data transfer speed is improved, but data integrity deteriorates
Solution Approach 1:
The transfer system is divided into a fast UDP path for initial data delivery and a separate reliability verification mechanism using descriptor files. This allows UDP to provide high speed while the descriptor-based tracking system ensures data integrity through systematic retransmission of missing blocks.
Solution Approach 2:
Descriptor files are prepared in advance to track expected data blocks. This preliminary tracking structure enables immediate identification of missing packets and facilitates rapid retransmission requests, maintaining both high speed and reliability.
3Reliability
If TCP acknowledgement mechanism is used, then data reliability is improved, but transfer efficiency deteriorates
Solution Approach 1:
The descriptor file serves as an intermediary that consolidates reliability tracking for multiple data blocks. Instead of individual acknowledgements for each packet, the descriptor file provides batch-level tracking and coordination, reducing the overhead of reliability mechanisms while maintaining data integrity.
Solution Approach 2:
The system creates and maintains a copy of the descriptor file that tracks the state of all data blocks. This copied tracking structure allows parallel processing of reliability verification without blocking the main data transfer stream, improving overall transfer efficiency.
4Speed
If UDP connection is used, then data transfer speed is improved, but data packet loss increases
Solution Approach 1:
The descriptor file provides continuous feedback about which data blocks have been successfully received and which are missing. This feedback mechanism enables the system to identify lost packets and trigger selective retransmissions, ensuring complete packet delivery while maintaining high transfer speeds.
Solution Approach 2:
The descriptor file is prepared in advance with all expected block identifiers, enabling immediate detection of missing packets upon arrival of data. This preliminary preparation eliminates delays in identifying and requesting retransmission of lost packets.
Data Source
AI summary
Systems and methods for transfer of data including establishing two separate connections, the two separate connections including a high speed connection and a high integrity connection. Blocks of data are exchanged over the high speed connection while the high integrity connection facilitates communication of descriptor data regarding data received over the high speed connection. As such, the data transfer speed of the high speed connection is utilized while communication via the high integrity connection allows for data reliability features not provided by the high speed connection. The systems and methods may include receiver side data handling such that data received chronologically out of order may be ordered at the receiver.