Structured Data Transfer Batching for High-Latency Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer protocols, such as MAPI, incur significant overhead and transmission delays when moving large datasets over high-latency and unreliable networks, inefficiently consuming local resources by requiring multiple operations for confirmation.
Innovation Solution
Implementing a structured data transfer optimization method that streams data in batches using asynchronous threaded processes, compressing data with algorithms like LZ, DEFLATE, and LZ-Reneau, and verifying integrity with checksums, allowing simultaneous reading and transmission of data blocks while reducing resource idling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transmitted through multiple operations with confirmation protocols, then data integrity is ensured, but transmission delays increase and productivity decreases
Solution Approach 1:
The patent segments large data blocks into smaller batches for transmission. Each batch is transmitted independently with its own confirmation protocol, allowing parallel processing and reducing overall transmission delay while maintaining data integrity through batch-level acknowledgments
Solution Approach 2:
The patent implements preliminary actions by pre-processing data into batches, pre-establishing transmission queues, and pre-configuring confirmation handlers before actual data transfer begins. This reduces overhead during active transmission and improves throughput
2Reliability
If multiple operations are performed for data transfer, then data reliability is improved, but local resources are consumed inefficiently
Solution Approach 1:
The patent maintains continuous useful action by implementing asynchronous processing where data preparation, transmission, and confirmation handling occur in parallel rather than sequential blocks. Threads continue processing subsequent batches while previous transmissions are being confirmed, eliminating idle resource periods
Solution Approach 2:
The patent introduces dynamic resource allocation where thread priorities and resource allocation adjust based on transmission state. Active transmission threads receive higher priority while waiting threads are deprioritized, optimizing resource utilization during different phases of the data transfer process
3Adaptability or versatility
If command and control information is transmitted with each data message, then protocol functionality is maintained, but overhead increases and transmission efficiency decreases
Solution Approach 1:
The patent extracts command and control information from individual data messages and consolidates it into batch-level headers and footers. This separation reduces per-message overhead while maintaining full protocol functionality at the batch level, where control commands are issued once per batch rather than repeatedly for each message
Solution Approach 2:
The patent merges multiple data messages into single transmission batches, combining their control information into unified batch headers. This consolidation eliminates redundant control commands for each individual message while preserving all necessary protocol functionality through the batch structure
Data Source
AI summary
Optimizations for data transmission may be provided. A portion of a data block may be read into a batch by a read thread on a first server. The batch may be passed to a transmission thread. The transmission thread may then transmit the first batch to a second server while the read thread asynchronously reads a second portion of the data block into another batch.