IIoT Reactive Buffering for Dynamic Data Batch Size Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Industrial Internet of Things (IIoT) data pipelines face data congestion and reliability issues due to the lack of prioritization and regulation strategies for data transfer, leading to potential data loss and increased recovery times when downstream nodes experience slow throughput or spikes in upstream data volume.
Innovation Solution
A reactive buffering system dynamically adjusts data batch sizes based on aggregated downstream metrics, such as current data processing latencies, to prevent overwhelming the pipeline and maintain data delivery reliability by implementing a request-response handshaking protocol and timestamp moderation for synchronized data delivery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data batches are sent without dynamic size adjustment, then data transfer simplicity is maintained, but data congestion occurs and reliability decreases
Solution Approach 1:
The system dynamically adjusts data batch sizes based on real-time downstream performance metrics. The batch size calculation component computes optimal batch sizes by considering downstream processing capacity and current pipeline conditions, transforming the static batch size approach into a dynamic adaptive system that resolves the contradiction between reliability and complexity.
Solution Approach 2:
The system implements a feedback mechanism where downstream nodes send performance metric data back to upstream nodes. The batch size calculation component uses this feedback to continuously optimize batch sizes, ensuring reliable data delivery while adapting to changing pipeline conditions. This closed-loop control resolves the contradiction by making the system responsive to actual performance.
2Productivity
If larger data batches are sent to improve throughput, then productivity increases, but data congestion occurs when downstream nodes experience slow throughput
Solution Approach 1:
The system dynamically adapts batch sizes based on real-time downstream performance. When downstream nodes experience slow throughput, the batch size calculation component reduces batch sizes to prevent congestion. When downstream capacity is high, larger batches are sent to maximize productivity. This dynamic adjustment resolves the contradiction between throughput and reliability.
Solution Approach 2:
The system changes the batch size parameter based on downstream performance metrics. The batch size calculation component adjusts this critical parameter in response to varying pipeline conditions, allowing the system to optimize throughput while maintaining reliability by preventing overload during low-capacity periods.
3Reliability
If data is queued without prioritization, then queue management simplicity is maintained, but data loss risk increases during congestion
Solution Approach 1:
The system segments the data queue into prioritized sections based on data importance and downstream capacity. The queue management component organizes queued data into different priority levels, allowing critical data to be transmitted first during congestion. This segmentation approach prevents data loss while maintaining manageable queue complexity through structured organization.
4Loss of time
If fixed batch sizes are used, then system simplicity is maintained, but recovery time increases when congestion occurs
Solution Approach 1:
The system dynamically adjusts batch sizes to accelerate recovery from congestion. When congestion is detected, the batch size calculation component reduces batch sizes to clear the backlog efficiently. When the pipeline recovers, larger batches resume to minimize downtime. This dynamic response significantly reduces recovery time compared to fixed batch sizes.
Solution Approach 2:
The system takes preliminary action to prevent and quickly resolve congestion. The batch size calculation component proactively adjusts batch sizes based on predicted downstream capacity, preventing congestion before it occurs. When congestion is anticipated, the system reduces batch sizes in advance, minimizing the need for lengthy recovery periods.
Data Source
AI summary
A reactive buffering system for use in IIoT data pipelines dynamically adjusts data accumulation and delivery by a node of a pipeline based on aggregated downstream metrics representing current data processing latencies of downstream nodes. Based on these downstream performance metrics, a reactive node that adjusts the size of the next data batch to be sent to an adjacent downstream node. The nodes of the data pipeline are configured to support a request-response based handshaking protocol whereby the nodes that send data to downstream nodes maintain up-to-date performance level information from adjacent downstream nodes. With this performance information, together with pipeline priorities, the sending node (or reactive node) adjusts the transmission rate and intermediate buffering of data. In this way, the nodes of the pipeline can dynamically regulate interim data storage to avoid overwhelming the pipeline system with too much data during periods of high latency.