Deep Learning Model Slicing for Edge Device Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to deploy large deep learning models with high accuracy, high availability, and extensibility without model compression, leading to inefficiencies in resource utilization and model deployment on edge devices.
Innovation Solution
The deep learning model is split into slices based on the capabilities of edge computing devices, with virtual models created and stored in a cache, allowing selection and deployment of model slices that match device capabilities, and monitoring device health to ensure stable operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the deep learning model is deployed without compression to maintain high accuracy, then the model accuracy is improved, but the device resource consumption (processor and memory capacity) increases significantly
Solution Approach 1:
The patent divides the large deep learning model into multiple smaller sub-models or slices that can be distributed across multiple edge devices. Each device runs a portion of the model, reducing the computational and memory burden on individual devices while maintaining overall model accuracy through coordinated inference across the distributed system.
2Measurement precision
If the neural network is made deeper to improve model accuracy, then the model accuracy is improved, but the model size increases significantly, increasing transfer and deployment time
Solution Approach 1:
The patent segments the large deep neural network into multiple smaller sub-networks that can be transferred and deployed independently to different edge devices. This segmentation reduces the transfer time for each sub-model while enabling the system to achieve the accuracy of the original large model through distributed computation.
Solution Approach 2:
The patent deploys only the necessary portions of the model to each edge device based on local needs and device capabilities, rather than transferring the complete large model to every device. This partial deployment approach reduces transfer time and resource consumption while maintaining adequate accuracy for local inference tasks.
3Use of energy by moving object
If model compression techniques (pruning or quantization) are applied to deploy on edge devices, then the device resource consumption is reduced, but the model accuracy decreases
Solution Approach 1:
Instead of compressing the model through pruning or quantization, the patent segments the full-precision model into distributed slices that run on multiple devices. This approach maintains the original model accuracy while reducing per-device resource consumption through parallel distributed inference.
4Measurement precision
If the deep learning model is made larger to improve accuracy, then the model accuracy is improved, but the model becomes difficult to relocate and balance for high availability
Solution Approach 1:
The patent divides the large model into smaller distributed slices that can be independently relocated and balanced across the edge device fleet. This segmentation enables flexible load balancing and high availability by allowing individual slices to be moved or replicated across devices based on runtime conditions, device availability, and inference demand.
Solution Approach 2:
The patent implements dynamic model slicing and distribution that adapts to changing device availability and inference workloads. Model slices can be dynamically relocated, scaled, or rebalanced across edge devices in response to runtime conditions, enabling high availability and load balancing for the distributed model system.
Data Source
AI summary
In an approach to deploying parallelizable deep learning models by adapting to the computing devices, a deep learning model is split into a plurality of slices, where each slice can exchange data with related slices. Virtual models are created from the plurality of slices, where the virtual models are based on capabilities of a plurality of devices on which the one or more virtual models are to be deployed, and further where each virtual model contains each slice of the plurality of slices. The one or more virtual models are stored in a cache. Responsive to determining that the deep learning model is to be deployed on one or more devices, a candidate model is selected from the virtual models in the cache, where the selection is based on information from a device monitor about the devices.