Dynamic Resource Scaling in Real-Time Stream Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Real-time stream processing systems face inefficiencies due to fluctuations in data flow, leading to wasted computing resources as they cannot automatically adjust computing node operations to match workload demands, requiring manual intervention and service disruptions for resource allocation.
Innovation Solution
A method and device that dynamically adjust computing resources by acquiring processing status, splitting message clusters, and assigning them to multiple computing units based on workload conditions, allowing for automatic augmentation or release of resources without service interruptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computing resources are allocated based on maximum capacity to ensure real-time processing during high peaks, then system reliability is improved, but computing resources are wasted during low data-processing periods
Solution Approach 1:
The system dynamically adjusts computing resources by monitoring data flow characteristics in real-time and automatically scaling computing nodes up or down based on actual workload demands, transforming the static resource allocation into a dynamic adaptive system that matches resource supply with actual processing needs
Solution Approach 2:
The system implements a feedback mechanism that continuously monitors data flow volume and processing status, using this information to automatically adjust the number of active computing nodes, creating a closed-loop control system that responds to changing workload conditions
2Productivity
If the number of computing nodes is adjusted manually to match workload fluctuations, then resource utilization is improved, but system complexity and operation difficulty increase
Solution Approach 1:
The system enables self-service by implementing automated monitoring and decision-making algorithms that independently determine when to scale computing nodes up or down based on predefined thresholds and current workload conditions, eliminating the need for manual operational intervention
3Adaptability or versatility
If computing nodes are added or removed to adjust computing resources, then system adaptability is improved, but system complexity increases due to topological structure changes
Solution Approach 1:
The system segments the computing task graph into independent, loosely-coupled micro-tasks that can be dynamically assigned to different computing nodes without requiring global topological reconfiguration, allowing individual node addition or removal without affecting the entire system structure
4Reliability
If message stream processing uses point-to-point synchronous transmission, then message transmission reliability is improved, but system adaptability decreases due to closely coupled upstream and downstream computing nodes
Solution Approach 1:
The system introduces a message queue as an intermediary component between upstream and downstream computing nodes, decoupling their direct synchronous connection while maintaining reliable message transmission through asynchronous queue-based communication that allows independent node operation
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for augmenting the capacity of computing resources in a real-time stream computing system is provided. In the system, computing tasks are transmitted by distributed message queues. The method includes determining whether the system includes a first computing unit having a workload exceeding pre-determined conditions; splitting a computing task transmitted through the distributed message queue and to be processed by the first computing unit that has a workload exceeding the pre-determined conditions, into a number of split computing tasks, and assigning the split computing tasks to a number of second computing units for processing, the number of second computing units corresponding to the number of split computing tasks.