Service Cluster Resource Adjustment for Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing auto-scaling schemes for service clusters in streaming data processing scenarios often lead to increased task instability or decreased resource utilization, resulting in unreasonable resource scaling.
Innovation Solution
A method and apparatus for resource adjustment in a service cluster, which determines resource adjustment information based on the task quantity of streaming data processing tasks, including a resource adjustment operation and a corresponding parallelism degree. This information is used to adjust the core quantity of central processor units and the quantity of computing nodes, thereby optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional auto-scaling schemes are used to adjust resources in service clusters, then resource utilization rate improves, but task stability deteriorates
Solution Approach 1:
The patent implements dynamic resource adjustment by continuously monitoring task traffic changes and dynamically scaling computing nodes and container cores. The system transitions from static resource allocation to dynamic scaling based on real-time traffic conditions, enabling the cluster to adapt to varying workloads while maintaining stability through controlled adjustment rates.
Solution Approach 2:
The patent changes key parameters including computing node quantity, container core quantity, and parallelism degree to optimize resource utilization. By adjusting these parameters based on traffic patterns and historical data, the system achieves better resource matching without causing excessive fluctuations that would harm task stability.
2Productivity
If resource scaling is increased to handle traffic spikes, then processing capability improves, but resource allocation reasonability deteriorates
Solution Approach 1:
The patent implements feedback mechanisms by monitoring task traffic changes, resource utilization rates, and parallelism degrees. The system uses this feedback to adjust resource allocation reasonsably, preventing over-scaling during traffic spikes and ensuring that resource increases are justified by actual demand patterns rather than causing wasteful allocation.
Solution Approach 2:
The patent performs preliminary actions by analyzing historical traffic data and predicting future resource needs before actual traffic spikes occur. This allows the system to pre-adjust resource allocation in a controlled manner, preparing the cluster for upcoming workload changes while maintaining allocation reasonability through data-driven decisions.
3Adaptability or versatility
If frequent resource adjustments are made to adapt to traffic changes, then adaptability improves, but system complexity increases
Solution Approach 1:
The patent implements periodic resource adjustment actions based on monitored traffic changes and predefined adjustment thresholds. Rather than continuous adjustment, the system performs periodic evaluations and adjustments at appropriate intervals, reducing the frequency of complex operations while maintaining adaptability to traffic patterns through systematic, scheduled optimization cycles.
Data Source
AI summary
The present disclosure provides a method and an apparatus of resource adjustment for a service cluster, an electronic device and a storage medium. The method includes: determining resource adjustment information for the service cluster according to a task quantity of streaming data processing tasks to be processed; determining a core quantity to be adjusted corresponding to central processor units and the second quantity of computing nodes to be adjusted, according to the parallelism degree to be adjusted, the first quantity of the computing nodes that have been created in the service cluster, and a resource utilization rate of central processor units in each computing node that has been created; and executing the resource adjustment operation according to the core quantity to be adjusted and the second quantity, and obtaining a service cluster subject to resource adjusting.