Data Processing Velocity Control for Stream Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data stream processing systems face challenges in managing non-uniform data stream velocities, leading to potential data loss and reduced reliability due to exceeding processing capabilities during peak data influx, especially in applications like financial services and telecommunications.
Innovation Solution
A data processing method that predicts traffic peaks using historical data, adjusts data receiving velocity, and employs cache queues to store excess data, ensuring that data is not lost during peak periods by reducing the data stream velocity and utilizing both receiving and sending cache queues to manage data flow effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the data stream velocity is increased to process more data, then the processing capability is improved, but data loss occurs when the velocity exceeds the maximum processing capability
Solution Approach 1:
The system performs preliminary actions by predicting future traffic peaks using historical data before they occur. The prediction module analyzes historical traffic patterns to forecast upcoming data surges, allowing the system to proactively adjust receiving velocity and utilize cache queues in advance, preventing data loss before it happens.
Solution Approach 2:
The system implements feedback mechanisms where the prediction module continuously monitors actual traffic against predicted traffic, and the receiving velocity is dynamically adjusted based on this feedback. The cache queue status also provides feedback to control the receiving velocity, creating a closed-loop control system that maintains reliability while optimizing throughput.
2Reliability
If the data receiving velocity is reduced to prevent data loss, then data reliability is improved, but the processing efficiency decreases
Solution Approach 1:
The system dynamically adjusts the data receiving velocity based on real-time conditions and predictions rather than maintaining a fixed velocity. The receiving velocity control module modifies the velocity adaptively according to predicted traffic peaks, current cache queue status, and actual processing capability, optimizing both reliability and efficiency under varying conditions.
Solution Approach 2:
The system changes the parameter of receiving velocity based on traffic conditions. During normal periods, the velocity is maintained at higher levels for efficiency. When traffic peaks are predicted or detected, the velocity parameter is reduced to prevent data loss, and cache queues are utilized to buffer the excess data.
3Reliability
If cache queues are used to store excess data, then data loss is prevented, but the system complexity increases
Solution Approach 1:
Cache queues serve as intermediary buffers between the data source and the processing operators. These queues decouple the receiving side from the processing side, allowing independent optimization of each. The prediction module and velocity control act as additional intermediaries that manage the flow through the cache queues, preventing data loss while maintaining manageable system complexity through modular design.
Data Source
Figure 1-a~1-b
Figure 2~3
Figure 4~5
AI summary
The present invention discloses a data processing method and a physical machine, so as to reduce a data loss in a process in which an executor processes data in a data stream. Embodiments of the present invention provide the data processing method, the method is applied to a physical machine in a stream system, the physical machine includes a first executor, and the method includes: predicting traffic of a to-be-processed data stream of the first executor in a first time period according to historical information about processing data by the first executor, so as to obtain prediction information of the traffic of the data stream in the first time period, where the historical information includes traffic information of data processed by the first executor in a historical time period, and the traffic prediction information includes predictors of traffic at multiple moments in the first time period; if the traffic prediction information includes a predictor that exceeds a threshold, reducing a data obtaining velocity of the first executor from a first velocity to a second velocity; and obtaining a first data set of the to-be-processed data stream at the second velocity.