ML Instance Recommendation via Latency Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Selecting an optimal computing instance configuration for providing inference services in cloud computing is challenging due to varying latencies and costs across different configurations, especially when handling unpredictable workloads.
Innovation Solution
A prediction machine learning model is trained to predict latency and other metrics for inferencing services executing on different computing instances and using various batch sizes, allowing for the selection of an optimal computing instance and batch size to minimize latency and cost.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple computing instance configurations are evaluated to find the optimal one, then accuracy of selection is improved, but resource consumption increases
Solution Approach 1:
The system performs preliminary evaluations of computing instance configurations during off-peak periods or beforehand, storing the results for later use. This allows the optimal configuration to be selected quickly during actual inference without re-evaluating all options, thus improving selection accuracy while reducing real-time resource consumption.
Solution Approach 2:
The system creates a predictive model that copies and generalizes performance patterns from evaluated configurations to unevaluated ones. By training on a subset of configurations and using the model to predict performance of other configurations, the system achieves accurate selection without exhaustively testing all options, reducing resource consumption while maintaining selection accuracy.
2Speed
If computing instance configuration is optimized for low latency, then processing speed is improved, but cost increases
Solution Approach 1:
The system dynamically adjusts parameters such as batch size and computing instance configuration based on workload characteristics and performance requirements. By changing these parameters adaptively, the system can achieve low latency when needed while using more cost-effective configurations during periods with lower latency requirements, thus balancing processing speed and cost.
Solution Approach 2:
The system implements dynamic configuration selection where the optimal computing instance and batch size are determined based on real-time or near-real-time conditions including workload characteristics, SLA requirements, and cost constraints. This dynamic approach allows the system to achieve low latency when necessary while reducing costs during periods with more flexible requirements.
3Productivity
If batch size is increased to improve resource utilization, then productivity is improved, but latency increases
Solution Approach 1:
The system dynamically adjusts batch size based on workload characteristics, SLA requirements, and performance monitoring. When latency constraints are strict, the system reduces batch size to meet latency requirements. When resource utilization is the primary concern and latency constraints are more flexible, the system increases batch size to improve productivity. This dynamic adjustment resolves the contradiction by adapting to different operational priorities.
Solution Approach 2:
The system changes the batch size parameter adaptively based on changing conditions and requirements. By adjusting this key parameter, the system can optimize for either resource utilization or latency depending on current priorities, thus resolving the contradiction between productivity and latency.
Data Source
AI summary
In various examples, a prediction machine learning model determines a set of computing instances capable of executing a machine learning model and a set of batch sizes associated with inferencing requests based on a set of model parameters associated with the machine learning model and a number of floating point operations (FLOPS). In such examples this information is used to update a user interface to indicate computing instances to perform inferencing operations.