ML Model Switchover With Memory Preloading and Score Calibration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model serving systems face challenges in seamlessly updating machine-learning models without causing service interruptions, schema incompatibilities, and score distribution disruptions, especially in high-availability computing environments, due to strong couplings between models and pipelines, requiring manual interventions and downtime.
Innovation Solution
A system and method for deploying and updating machine-learning models in a distributed computing environment that uses memory preloading and dynamic configuration handling to ensure seamless model updates, maintaining schema compatibility and score consistency, allowing uninterrupted service delivery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual model replacement is performed in traditional model serving systems, then model updates can be implemented, but service interruptions and downtime occur
Solution Approach 1:
The system performs preliminary actions by pre-loading the new model into memory before the current model needs to be replaced. This allows the new model to be ready and waiting in memory, so when replacement is triggered, the switch can occur immediately without service interruption or downtime.
2Measurement precision
If model schema is changed to improve model performance, then model accuracy can be enhanced, but schema incompatibility issues arise
Solution Approach 1:
The system introduces an intermediary schema translation layer that sits between the new model with updated schema and the existing data pipelines. This intermediary automatically translates and adapts data between different schema versions, allowing the system to benefit from improved model accuracy while maintaining compatibility with existing infrastructure without requiring manual schema synchronization.
3Stability of the object's composition
If model calibration is performed offline to ensure score distribution consistency, then score distribution stability is maintained, but deployment time increases
Solution Approach 1:
The system performs model calibration in advance during offline preprocessing, before the model is deployed to production. This preliminary calibration ensures that the model's score distribution is already optimized and consistent with business requirements, eliminating the need for time-consuming calibration processes during deployment and allowing for rapid model replacement.
Data Source
AI summary
Computer-implemented method and system for deployment of a first machine-learning model, and replacement, without service interruption, of a second machine-learning model in active on-line use, comprising: receiving, at a controller, a replacement request; in response to said replacement request, triggering the deployment of the first model and triggering the calculation of features to be used, collecting output data from the first model, fitting and inserting one or more calibration functions downstream from the first model, and routing inference requests to the first model instead of the second model; wherein the triggered deployment of the first model comprises preloading the first model into CPU or GPU memory, and making available the calculated features and the preloaded first model by CPU or GPU, respectively, before the inference requests are routed to the first model, thus enabling that no additional latency is added when traffic is rerouted to the first model.