Incremental ML Model Deployment Reduces Inference Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in deploying pre-trained machine learning (ML) models efficiently over networks, leading to high inference startup latency due to the large size of ML models and the time required for their deployment.
Innovation Solution
The proposed solution involves deploying ML models in an incremental fashion from service provider infrastructure to devices over a network, using a bit incremental model delivery method. This approach allows for the initial deployment of a lower precision model, followed by incremental updates to higher precision models, enabling continuous improvement of model precision without affecting ongoing inference operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a full precision machine learning model is deployed immediately, then model accuracy and performance are improved, but inference startup latency increases significantly
Solution Approach 1:
The machine learning model is segmented into multiple precision levels (e.g., low precision and high precision versions). The system initially deploys a low precision model that can be quickly loaded, then progressively updates it to high precision through incremental model updates. This segmentation allows the model to be deployed in parts rather than requiring the complete high precision model upfront, thereby reducing startup latency while maintaining the option to achieve high precision when needed.
Solution Approach 2:
A low precision model is deployed preliminarily before the high precision model is fully available. This preliminary deployment allows the inference system to start operating immediately with acceptable accuracy, and then gradually transition to the high precision model through incremental updates. The preliminary action of deploying a lighter model first eliminates the blocking delay that would otherwise occur while waiting for the complete high precision model to be downloaded and loaded.
2Measurement precision
If incremental model updates are applied during inference, then model precision improves over time, but system complexity increases
Solution Approach 1:
The incremental model update mechanism enables continuous improvement of model precision without interrupting the inference service. Updates are applied while the system remains operational, allowing the model to evolve from low precision to high precision in the background. This continuity ensures that the useful action of serving requests is never stopped, while the model gradually improves its precision through successive updates.
Solution Approach 2:
The system dynamically adjusts the model precision level based on operational needs and update availability. The model can transition between different precision states (low precision operational state and high precision operational state) depending on whether updates are available and when they are applied. This dynamic behavior allows the system to optimize between speed (using low precision models) and accuracy (using high precision models) based on real-time conditions.
Data Source
AI summary
In accordance with example embodiments of the invention there is at least a method and apparatus to perform executing a machine learning inference loop of a currently deployed or stored at least one machine learning model, wherein the currently deployed or stored at least one machine learning model is identified based on a manifest file received from a communication network; based on determined factors, requesting from the communication network a model update to trigger the model update for use with the currently deployed or stored at least one machine learning model; based on the request, receiving information from the communication network comprising the model update; and based on the information, performing a model update to update the currently deployed or stored at least one machine learning model. Further, receiving, based on determined factors, from a user equipment a communication to trigger a machine learning model update for use with a currently deployed or stored at least one machine learning model at the user equipment; based on the communication, determining information comprising the model update; based on the determining, sending towards the client the information comprising the model update for a model update to update the currently deployed or stored at least one machine learning model.