ML Model Switchover with Memory Preloading and Output Calibration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model serving systems face challenges in seamlessly updating machine-learning models in active use without causing service interruptions, particularly in high-availability environments, due to issues with schema evolution and score distribution changes, which are exacerbated in distributed computing scenarios.
Innovation Solution
A system and method for deploying and replacing machine-learning models in a shadow mode, utilizing memory preloading and dynamic configuration handling to ensure seamless updates without service interruptions, by preloading new models into CPU or GPU memory and calibrating their outputs to match the old model's distribution, ensuring compatibility and minimal impact on downstream systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a new machine-learning model is deployed to replace an existing model in active use, then model performance and accuracy can be improved, but service interruption and latency increases occur
Solution Approach 1:
The system pre-loads the new model into memory before it is needed for service, so that when the model replacement is triggered, the new model is already ready and can be activated immediately without service interruption. This preliminary preparation eliminates the downtime that would normally occur during model deployment.
Solution Approach 2:
The system uses a model proxy or gateway component that mediates between the client and the actual model. During model replacement, the proxy can route requests to either the old or new model, allowing seamless switching without affecting service availability. This intermediary layer decouples the client from the model deployment process.
2Adaptability or versatility
If model updates are performed in distributed computing systems, then model scalability and flexibility are improved, but system complexity and coordination overhead increase
Solution Approach 1:
The system divides the model serving functionality into separate, independent components: model loading, model execution, and model management. Each component can be independently deployed, updated, and scaled. This segmentation allows the system to handle model updates in a distributed environment without increasing overall system complexity, as each segment manages its own state and coordination.
3Stability of the object's composition
If model output calibration is performed to match score distributions, then downstream system compatibility is maintained, but additional processing time and computational overhead are introduced
Solution Approach 1:
The system performs calibration data collection and analysis in advance, building calibration models before they are needed for actual model replacement. This preliminary calibration preparation ensures that when model replacement occurs, the calibration process is already optimized and can be applied quickly, minimizing the time added to the deployment process.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Computer-implemented method and system for deployment of a first machine-learning model, and replacement, without service interruption, of a second machine-learning model in active on-line use, comprising: receiving, at a controller, a replacement request; in response to said replacement request, triggering the deployment of the first model and triggering the calculation of features to be used, collecting output data from the first model, fitting and inserting one or more calibration functions downstream from the first model, and routing inference requests to the first model instead of the second model; wherein the triggered deployment of the first model comprises preloading the first model into CPU or GPU memory, and making available the calculated features and the preloaded first model by CPU or GPU, respectively, before the inference requests are routed to the first model, thus enabling that no additional latency is added when traffic is rerouted to the first model.