Message-Based Data Transport Protocol for Large-Scale Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data backup and transfer technologies, such as TCP, are inefficient for large-scale data transport due to the decomposition of data into small packets and stop-and-go behavior, which hampers the continuous delivery of gigabytes or terabytes of data across networks.
Innovation Solution
A message-based scalable data transport system that uses a communication engine to decompose data into message objects, buffer them over connections, and utilize multiple connections for continuous delivery, optimizing channel utilization and eliminating the need for acknowledgment of individual packets, thereby achieving high throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If TCP protocol is used for data transport, then data can be transmitted reliably, but the data is decomposed into small packets causing inefficient transport for large-scale data backup
Solution Approach 1:
The invention segments data into message objects of variable size (not fixed small packets) that can be efficiently transmitted over the network. The communication engine creates message objects that balance reliability with transport efficiency, allowing larger data units to be transmitted without the stop-and-go behavior caused by TCP's small packet granularity.
Solution Approach 2:
The communication engine acts as an intermediary layer between the data source and TCP protocol. It buffers data into message objects and manages the interaction with TCP, allowing the underlying TCP protocol to provide reliability while the message-based approach optimizes for productivity by reducing the stop-and-go behavior.
2Ease of operation
If TCP flow control is implemented, then data transmission can be controlled, but the system exhibits stop-and-go behavior reducing continuous delivery capability
Solution Approach 1:
The communication engine maintains continuous data delivery by buffering message objects and continuously pushing them through the TCP connection. The engine decouples the flow control mechanism from the data delivery, allowing TCP to manage its own flow while the message-based system maintains continuous progression without stop-and-go behavior.
Solution Approach 2:
The communication engine performs preliminary buffering of data into message objects before transmission. This preliminary action allows the system to prepare data in advance and continue delivering it smoothly, reducing the impact of TCP's flow control pauses and maintaining more continuous delivery speed.
3Productivity
If data is transmitted using conventional TCP protocol, then basic data transfer is achieved, but channel utilization is not maximized for large-scale transfers
Solution Approach 1:
The communication engine dynamically adjusts message object sizes and transmission timing based on channel conditions. It can optimize the balance between message size and transmission speed, maximizing channel utilization by adapting to network conditions rather than using fixed small packets or static transmission patterns.
Solution Approach 2:
The system changes key parameters such as message object size, buffer dimensions, and transmission timing to optimize channel utilization. By varying these parameters dynamically, the system achieves higher throughput and better channel efficiency for large-scale data transfers compared to conventional fixed TCP parameters.
Data Source
AI summary
The invention deploys a message-based data transport platform, in which a communication engine controls the delivery of server or other backup data or other large quantity data to a remote site based on message objects as fundamental units. Data sessions from one or more servers or other data sources may be buffered over established pipes for delivery to a remote data host which likewise contains messaging control. Relatively large message units may wait for communication over the channel in a queue, and be released when prior messaging is complete. Completion port or other polling or other mechanisms may indicate that connections have become open for use. In embodiments the message objects may be signed, or encrypted to safeguard against alteration or unauthorized viewing. The communication engine may rely upon TCP or other underlying transport mechanisms to effect lower level transmission.