Erasure-Coded UDP Data Transfer for Low-Latency Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transfer methods, such as TCP, face challenges in low-latency and low-data-loss transmission, especially in reliable networks, while UDP lacks data protection against loss, necessitating a more efficient solution for large data transfers in web-based applications and cloud computing.
Innovation Solution
Implementing data-redundancy encoding techniques, like erasure coding, before transmission over a UDP connection, which adds redundant information to the data, allowing reconstruction even if some packets are lost, thereby achieving low-latency and reliable data transfer without the overhead of traditional TCP methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If TCP connections are used for reliable data transfer, then data reliability is improved, but transmission latency and overhead increase
Solution Approach 1:
The patent segments data into fixed-size blocks and applies erasure coding to generate multiple encoded blocks. This allows the data to be divided into independent units that can be transmitted separately over UDP, achieving reliability through redundancy rather than through TCP's connection-oriented reliability mechanisms, thus reducing latency and overhead.
Solution Approach 2:
The patent performs erasure coding preprocessing on the data blocks before transmission. By adding redundant encoded blocks in advance, the system ensures that sufficient data can be reconstructed even if some packets are lost, eliminating the need for TCP's retransmission protocols and reducing transmission latency.
2Speed
If UDP protocol is used for low-latency transmission, then transmission speed is improved, but data loss tolerance increases
Solution Approach 1:
The patent applies erasure coding to create redundant encoded blocks before transmission. These redundant blocks act as a cushion against packet loss, allowing the receiver to reconstruct the original data even if some encoded blocks are lost during transmission, thus providing data protection while maintaining UDP's low-latency characteristics.
Solution Approach 2:
The patent changes the reliability parameter by transforming the data into encoded form with built-in redundancy. This parameter change allows the system to achieve reliability guarantees typically associated with TCP but using UDP's faster transmission mechanism, effectively decoupling reliability from transmission protocol choice.
3Reliability
If TCP overhead mechanisms are applied, then error correction is improved, but device complexity and resource usage increase
Solution Approach 1:
The patent extracts the error correction functionality from the transport protocol layer and implements it independently through erasure coding. This separates the reliability mechanism from the transport protocol, allowing UDP to handle transmission while erasure coding handles error correction, thereby reducing overall system complexity and resource usage.
Solution Approach 2:
The patent creates multiple copies of encoded data blocks through erasure coding. Instead of using TCP's complex acknowledgment and retransmission logic, the system simply transmits multiple encoded copies, allowing the receiver to reconstruct the original data from any sufficient subset of received blocks, significantly simplifying the error correction mechanism.
4Productivity
If data is transmitted without redundancy encoding, then transmission efficiency is improved, but data loss increases
Solution Approach 1:
The patent applies partial redundancy encoding by generating a specific number of encoded blocks where only a subset is needed for reconstruction. This excessive action of creating more encoded blocks than strictly necessary ensures that sufficient data is available for reconstruction while maintaining transmission efficiency, as the receiver only needs to obtain a threshold number of blocks.
Data Source
AI summary
A system for transmitting data over a network may include data processed according to a data-redundancy encoding technique such as erasure coding to be transmitted via an unreliable, connectionless transmission protocol, for example user datagram protocol (“UDP”). A transmitting manager may receive a request from an application to transmit data. The transmission manager may select encoding parameters and encode data to be transferred using a data-redundancy encoding technique. The transmission manager may initiate transmission of the encoded data and payload header data may via a network, for example via a network communication stream or protocol stack. A reconstruction manager associated with the receiving node may reconstruct the original data using the received erasure coding data and payload header data. In some embodiments, the receiving node may transmit telemetry data to the erasure coding algorithm. The erasure coding algorithm may use received telemetry data to adjust parameters of the erasure coding algorithm for future erasure coding and transmission operations. In some systems, telemetry data may include a request to retransmit all or a portion of the erasure coded data.