Elastic Scaling Plug-in for Distributed Data Flow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing resource elastic scaling controllers for distributed data flows face inefficiencies in accurately and quickly adjusting resource allocation to match fluctuating data load conditions, often requiring multiple iterations and resulting in suboptimal performance due to factors like network overheads and heterogeneous machine capabilities.
Innovation Solution
A plug-in for resource elastic scaling that integrates a decision maker, a decision model, and a scaling operation sample library, using machine learning to predict and refine resource allocation decisions, and providing them to the scaling controller through HTTP interfaces, allowing for adaptive and efficient resource adjustments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a computing-based controller is used to calculate optimal instance quantity by monitoring traffic of each node, then the controller can faster achieve elastic resource scaling compared to rule-based controllers, but multiple attempts are still required to complete one time of resource elastic scaling due to the need for iterative calculation and verification
Solution Approach 1:
The patent pre-calculates and stores the relationship between data inflow rates and optimal instance quantities in a lookup table during system initialization or offline training. When scaling is needed, the controller directly queries this pre-computed table based on current traffic conditions, eliminating the need for iterative calculations and multiple attempts during runtime scaling operations
Solution Approach 2:
The patent prepares scaling decisions in advance by creating a mapping between different data load scenarios and optimal instance configurations. This pre-prepared knowledge base acts as a cushion that allows the system to respond immediately to scaling requirements without undergoing repeated trial-and-error adjustments
2Reliability
If instance quantity is increased to match sudden increase in data flow load, then the processing rate can meet the data inflow rate, but the distributed data flow may occupy too many resources when load decreases, resulting in resource waste
Solution Approach 1:
The patent implements dynamic instance quantity adjustment by continuously monitoring data inflow rates and querying the pre-computed lookup table to determine the optimal instance count. This dynamic approach allows the system to scale instances up or down in real-time based on actual load conditions, ensuring processing rate consistency during high load while minimizing resource occupation during low load periods
Solution Approach 2:
The patent employs feedback mechanisms where the controller continuously monitors current data flow load and processing performance, then uses this feedback to query the lookup table and adjust instance quantities accordingly. This closed-loop control ensures the system maintains optimal resource allocation by responding to actual system state rather than using static or over-provisioned configurations
Data Source
AI summary
The present disclosure relates to a plug-in for enhancing resource elastic scaling of a distributed data flow and a method for enhancing a plug-in for enhancing resource elastic scaling of a distributed data flow. The plug-in is connected to a scaling controller used for resource elastic scaling of a distributed data flow. The plug-in includes a decision maker, a decision model, and a scaling operation sample library. The scaling controller registers a data flow to the plug-in through a first interface. The scaling controller sends an optimal decision of resource scaling in each status to the plug-in through a second interface. The scaling operation sample library is configured to record the optimal decision of resource scaling in each status. The decision model is configured to predict a received data flow based on the optimal decision recorded in the scaling operation sample library, to generate a prediction decision.