Learned Demand Prediction for Resource-Aware Microservice Throttling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing microservice management techniques in clustered computing systems face inefficiencies due to inaccurate or static predictions about future demands, leading to excessive resource consumption or latency, particularly in dynamic and stochastic environments.
Innovation Solution
Implementing a continuously trained microservices lifecycle predictive model that distinguishes between short-term and long-term demand predictions, using unsupervised learning to dynamically adjust microservice admission and resource allocation based on actual cluster conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If eager loading is used to pre-load microservices for potential future calls, then service availability is improved, but resource consumption increases unnecessarily
Solution Approach 1:
The system dynamically adjusts microservice loading strategy based on predicted demand. The predictive model continuously learns from historical data to determine whether to eager-load or lazy-load each microservice, making the loading strategy adaptive rather than static. This resolves the contradiction by making resource allocation dynamic - pre-loading only when prediction indicates high probability of need, and lazy-loading when prediction indicates low probability.
Solution Approach 2:
The system changes the parameter of loading timing based on predictive model outputs. Instead of using a fixed eager-loading or lazy-loading parameter, the system adjusts the loading timing parameter dynamically based on learned demand patterns, historical data, and predicted future calls. This allows optimization of both availability and resource consumption by selecting the appropriate loading parameter for each specific case.
2Quantity of substance
If lazy loading is used to defer microservice loading until actual calls, then resource consumption is reduced, but start-up latency increases
Solution Approach 1:
The system dynamically selects between eager-loading and lazy-loading based on predictive model outputs. When the model predicts high demand probability, it switches to eager-loading to avoid latency. When prediction indicates low demand probability, it uses lazy-loading to conserve resources. This dynamic selection resolves the contradiction by adapting the loading strategy to predicted actual usage patterns.
Solution Approach 2:
The system performs preliminary action (eager-loading) only when the predictive model indicates high probability of imminent use. By using historical data and learned patterns to predict future calls, the system can proactively load microservices in advance of actual calls when beneficial, thereby avoiding start-up latency without unnecessarily consuming resources for microservices that won't be used.
3Device complexity
If static demand predictions are used for microservice management, then system complexity is reduced, but accuracy of predictions deteriorates
Solution Approach 1:
The system uses self-service through automated machine learning models that continuously learn from historical data without requiring complex manual configuration. The predictive model automatically improves its accuracy over time by learning from patterns in the data, resolving the contradiction by providing high prediction accuracy through automated learning rather than complex static rules requiring manual maintenance.
Solution Approach 2:
The system implements feedback loops where prediction outcomes are continuously monitored and used to retrain and improve the predictive model. Historical data from actual microservice usage patterns feeds back into the learning system, continuously improving prediction accuracy. This feedback mechanism maintains high accuracy without requiring increasingly complex static prediction rules.
4Measurement precision
If continuous training of predictive models is implemented, then prediction accuracy is improved, but processing demands increase
Solution Approach 1:
The system uses periodic action by training the predictive model at appropriate intervals rather than continuously in real-time. The model is trained on historical data batches periodically, balancing the need for accurate predictions with the computational cost of training. This resolves the contradiction by providing sufficiently accurate predictions through periodic training rather than exhaustive continuous training that would consume excessive processing power.
Data Source
AI summary
A dynamically-managed microservices platform. The microservices platform is configured to selectively accept admission of microservices and to selectively throttle microservices based on a continually-trained machine learning model. The system is configured to implement multiple microservice lifecycle strategies, where individual ones of the multiple microservice lifecycle strategies use a prediction model having long-term and short term demand predictions. One or another of the multiple microservice lifecycle strategies are invoked at different times under different conditions. Under a first set of conditions, a prediction model is used to select a first strategy to admit or throttle a microservice based upon long-term or short-term demand predictions. Under a second set of conditions, the prediction model is used to select a different strategy to admit or throttle a microservice. The prediction model is continually trained such that new sets of observations and new sets of predicted outputs are codified into the machine learning model.