Coded Stream Partitioning With Parity Recovery for Slow Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream processing systems face challenges in error correction and latency management due to failures or slow processing of processors, leading to inefficient network bandwidth and computing resource consumption.
Innovation Solution
A system that partitions streaming data into data chunks and generates parity chunks using algorithms like Reed-Solomon, allowing the system to recover missing data by utilizing parity chunks, thereby tolerating a defined quantity of failed or slow processors without significant latency or interruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system waits for all data chunks to be received before processing, then data integrity is maintained, but processing latency increases significantly
Solution Approach 1:
The system performs preliminary actions by generating and transmitting parity chunks along with data chunks before all data is received. This allows the receiving system to begin processing and reconstruction operations in advance, reducing overall latency while maintaining data integrity through pre-established error correction capabilities
Solution Approach 2:
Parity chunks serve as intermediary elements that mediate between transmitted data and the final reconstructed output. These intermediate parity representations enable partial processing and error recovery without requiring complete data reception, thus reducing latency while preserving integrity
2Reliability
If the system uses error correction codes and parity chunks to tolerate processor failures, then reliability improves, but system complexity increases
Solution Approach 1:
The system segments data into discrete chunks and applies error correction independently to each chunk using parity generation. This modular approach distributes the complexity of fault tolerance across multiple independent units rather than requiring a complex centralized error correction system, making the overall system more manageable while maintaining high reliability
Solution Approach 2:
The system changes parameters by using simple parity bit appending and XOR operations instead of complex error correction algorithms. This parameter simplification maintains fault tolerance capabilities while significantly reducing computational complexity and system overhead
3Reliability
If the system retransmits missing data chunks, then data completeness is ensured, but network bandwidth consumption increases
Solution Approach 1:
Parity chunks act as intermediaries that can be transmitted instead of original data chunks when errors occur. These parity representations serve as substitutes that enable data recovery without requiring retransmission of the original chunks, thus reducing network bandwidth consumption while ensuring data completeness
Solution Approach 2:
The system discards the need to retransmit original missing chunks by using parity information for recovery. Instead of recovering data through retransmission, the system recovers missing chunks by computing them from received data and parity chunks, significantly reducing network bandwidth usage
Data Source
AI summary
Described herein is a system and method for coded streaming data to facilitate recovery from failed or slow processor(s). A batch of processing stream data can be partitioned into a plurality of data chunks. Parity chunk(s) for the plurality of data chunks. The plurality of data chunks and the parity chunk(s) can be provided to processors for processing. Processed data of at least some (e.g., one or more) of the plurality of data chunks, and, processed data of parity chunk(s) are received. When it is determined that processed data for a pre-defined quantity of data chunks has not been received by a pre-defined period of time, the processed data for particular data chunk(s) of particular processor(s) from which processed data has not been received are determined based, at least in part, upon the received processed parity chunk(s) and the received processed data chunk(s).