Neural Network Partition Updates for Seamless On-Device Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Updating neural network models on computation devices poses challenges in maintaining model responsivity during the update process, as conventional methods require downtime and interrupt inference operations.
Innovation Solution
The neural network model is updated in partitions, with the selection of partition size and order optimized to ensure seamless execution during download, allowing the model to operate on input data and provide output data before completion of the update process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the neural network model is updated on a computation device using conventional methods, then the model can be updated to handle new scenarios, but the model must be taken offline causing downtime and interrupting inference operations
Solution Approach 1:
The neural network model is divided into multiple partitions that can be downloaded and updated independently. This segmentation allows the computation device to download model partitions in parallel with ongoing inference operations, eliminating the need to take the entire model offline for updates.
Solution Approach 2:
Model partitions are downloaded in advance during periods when computation capacity is available, before they are needed for inference. This preliminary action ensures that updated model partitions are ready and can be seamlessly switched to without interrupting service.
2Reliability
If the entire neural network model is downloaded for updating, then the model can be fully updated, but the download time increases the end-to-end update time and reduces responsiveness
Solution Approach 1:
The model update process is segmented into multiple smaller partition downloads rather than one large download. This allows the system to download, process, and switch to partitions incrementally, reducing the perceived update time and maintaining responsiveness throughout the process.
Solution Approach 2:
The download and update process continues in parallel with inference operations on the current model version. The computation device maintains continuous useful action by serving inference requests with the existing model while simultaneously downloading and preparing updated partitions in the background.
3Productivity
If the neural network model is updated sequentially to maintain responsivity, then downtime is minimized, but the update process becomes complex and computation device capacity is not optimally utilized
Solution Approach 1:
The model is segmented into partitions that can be managed independently with simple download-and-switch logic. This segmentation simplifies the update process compared to managing entire model versions, as each partition can be downloaded and activated independently without complex coordination.
Solution Approach 2:
The system dynamically allocates computation device capacity between running inference on the current model version and downloading/processing updated partitions. This dynamic utilization optimizes resource usage by adapting to available computation capacity rather than following a fixed sequential update schedule.
Data Source
AI summary
A method for updating a neural network model on a computation device. The method includes estimating a bandwidth for data download from a server device to the computation device; estimating a time point of available computation capacity of the computation device; computing a maximum partition size as a function of the bandwidth and the time point; and causing download of a selected partition of the neural network model from the server device to the computation device, the selected partition being determined based on the maximum partition size. The method enables the selected partition to be executed by the computation device upon downloading and thereby provides for seamless updating of the neural network model while it is being executed on the computation device.