Neural Network Model Partitioning for Edge Device Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Resource-constrained devices face challenges in deploying neural network models due to memory and processing limitations, leading to compromised accuracy and efficiency when quantization or pruning is applied, and require external cloud services that raise privacy concerns and incur latency.
Innovation Solution
The method involves partitioning a neural network model into lightweight sub-models based on constraint information, allowing each sub-model to be loaded and executed within the device's memory limits, with intermediate results passed between sub-models to maintain accuracy and perform computations locally, while ensuring privacy by not transmitting sensitive data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the DNN model is quantized or pruned to reduce memory requirements, then the model can be deployed on resource-constrained devices, but the accuracy or efficiency of the DNN model is compromised
Solution Approach 1:
The patent divides the DNN model into multiple sub-models that can be loaded and executed sequentially or in parallel. Each sub-model contains a portion of the original model's layers and parameters, allowing the system to process inputs through multiple smaller computations rather than loading the entire large model into memory. This segmentation enables deployment on resource-constrained devices while maintaining the full model's accuracy by preserving all original weights and activations across the sub-models.
2Adaptability or versatility
If the DNN model size is increased to perform more complex machine learning tasks, then the capability and accuracy of the device improve, but the memory and processing requirements exceed the constraints of edge devices
Solution Approach 1:
The patent enables edge devices to run complex DNN models by segmenting them into smaller sub-models that fit within device memory constraints. Each sub-model processes a portion of the computation, and intermediate results are passed between sub-models to achieve the same functionality as the full model, thus enabling complex ML tasks on resource-constrained devices.
Solution Approach 2:
The patent introduces a temporal dimension to model execution by loading and unloading sub-models sequentially over time. Instead of requiring all model parameters to reside in memory simultaneously, the system loads only the necessary sub-model into memory when needed, executes the computation, and then unloads it, effectively trading time for memory space.
3Adaptability or versatility
If data is transmitted to external cloud services for processing, then complex machine learning tasks can be performed, but privacy concerns arise and latency is incurred
Solution Approach 1:
The patent enables edge devices to perform complex machine learning inference tasks autonomously by executing partitioned DNN models locally on the device. The sub-models are deployed to and run on the edge device itself, eliminating the need to transmit data to external cloud services for processing. This self-service approach maintains full model capability while reducing latency and preserving data privacy.
Data Source
AI summary
A method for deploying neural network models on resource-constrained devices is provided. The method includes storing a model file that includes a neural network model and determining constraint information associated with deployment of the neural network model on the electronic device. The method further includes determining a partition of the neural network model based on the constraint information and the model file and extracting sub-models from the neural network model based on the partition. The method further includes receiving an input associated with a machine learning task and executing operations for loading a sub-model in a working memory of the electronic device, applying the sub-model on the input to generate an intermediate result, and unloading the sub-model from the working memory. The method further includes executing the operations for a next sub-model to generate an output and rendering the output. The intermediate result is an input for the next sub-model.