Distributed Stream Computing Back Pressure Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed stream computing systems face back pressure issues due to a higher data reception rate than processing rate, leading to resource exhaustion and potential data loss, which is currently managed by temporarily shutting down data streams, affecting all computing nodes and efficiency.
Innovation Solution
A processing method that dynamically adjusts the data sending rate by introducing and removing marks based on data ratio thresholds, allowing upstream computing nodes to decrease or increase the flow data sending rate, thereby managing back pressure and improving system efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the data sending rate is increased to improve processing efficiency, then productivity is improved, but back pressure occurs causing resource exhaustion and data loss
Solution Approach 1:
The patent implements a feedback mechanism where downstream computing nodes monitor their processing capacity and send control signals (marks) upstream when back pressure is detected. The upstream nodes receive these feedback signals and dynamically adjust their data sending rates accordingly, creating a closed-loop control system that prevents resource exhaustion while maintaining high productivity.
Solution Approach 2:
The patent makes the data sending rate dynamic rather than static. Upstream computing nodes continuously adjust their sending rates based on real-time feedback from downstream nodes. The sending rate increases when downstream capacity is available and decreases when back pressure is detected, allowing the system to adapt to changing conditions and maintain optimal performance without data loss.
2Reliability
If data streams are temporarily shut down to resolve back pressure, then reliability is improved, but productivity of all computing nodes deteriorates
Solution Approach 1:
The patent applies local quality control where only the specific upstream computing node experiencing back pressure adjusts its sending rate, while other nodes continue operating at full capacity. The control signal is targeted and localized to the affected node rather than shutting down all data streams system-wide, maintaining overall productivity while ensuring system stability.
Solution Approach 2:
Instead of static shutdowns, the patent implements dynamic rate adjustment. The upstream node gradually modulates its sending rate based on real-time feedback, allowing the system to maintain stability through continuous adaptation rather than abrupt shutdowns. This dynamic approach preserves productivity by keeping nodes operational at optimized rates rather than completely stopping data flow.
3Device complexity
If a fixed data sending rate is used to simplify control, then device complexity is reduced, but back pressure management becomes inefficient
Solution Approach 1:
The patent introduces a feedback-based control mechanism where downstream nodes monitor their processing capacity and send control marks upstream when needed. This feedback loop enables efficient back pressure management by automatically adjusting sending rates based on real-time system state, improving productivity without requiring complex centralized control or manual intervention.
Solution Approach 2:
The system implements self-service control where computing nodes autonomously adjust their own sending rates based on feedback from downstream nodes. Each node independently monitors and regulates its data flow without requiring external control, simplifying the overall control architecture while maintaining efficient back pressure management through distributed autonomous decision-making.
Data Source
AI summary
A processing method based on distributed stream computing is performed by a computing device. After receiving flow data sent by an upstream computing node, the computing device stores the flow data into a flow data pool. The computing device collects a data ratio of the flow data in the flow data pool to a total capacity of the flow data pool. When the collected data ratio is greater than or equal to a first threshold, the computing device performs an operation of adding a mark for enabling the upstream computing node to decrease a flow data sending rate; when the collected data ratio is less than or equal to a second threshold, the second threshold being less than the first threshold, the computing device performs an operation of deleting the mark for enabling the upstream computing node to increase the flow data sending rate.