Neural Network Model Extraction for Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face challenges in executing real-time inference processing on edge devices due to high computational demands and large memory requirements, making it difficult to deploy them on resource-constrained devices like mobile or in-vehicle systems without significant latency and high communication costs.
Innovation Solution
A providing apparatus that includes a scalable neural network model capable of changing its calculation amount by decomposing tensors of weighting coefficients, allowing for the extraction of smaller machine learning models tailored to specific device hardware specifications, which are then deployed to edge devices for efficient inference processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a deep neural network with many layers and high computational power is used, then recognition performance is improved, but the device requires large memory capacity and high computational resources, making it unsuitable for edge devices
Solution Approach 1:
The patent segments the deep neural network into multiple functional modules including an extraction unit that divides the network into a first neural network for feature extraction and a second neural network for classification. This segmentation allows the heavy computational burden to be distributed, with the first network performing lightweight feature extraction on edge devices while the second network handles more complex processing, thus reducing the memory and computational requirements on resource-constrained devices while maintaining overall recognition performance
Solution Approach 2:
The patent applies local quality by assigning different functional characteristics to different parts of the neural network system. The first neural network is optimized for efficiency and low computational requirements to operate on edge devices, while the second neural network is optimized for accuracy and can operate on more powerful servers. This localized optimization allows each component to perform its specific function with appropriate resource allocation, resolving the contradiction between overall performance and local resource constraints
2Speed
If a deep neural network is deployed on edge devices, then real-time inference is enabled, but the computational demands and memory requirements exceed the capabilities of resource-constrained devices
Solution Approach 1:
The patent segments the inference process into two stages: a lightweight feature extraction stage performed in real-time on edge devices using the first neural network, and a more computationally intensive classification stage performed on servers using the second neural network. This segmentation enables real-time response on edge devices for critical functions while offloading heavy computation to servers, thus achieving real-time inference capability without exceeding the computational and memory constraints of edge devices
Solution Approach 2:
The patent applies partial action by having the first neural network on edge devices perform only feature extraction rather than complete inference. This partial processing achieves real-time response for the most time-critical aspect of inference, while the remaining classification computation is performed partially or excessively on servers with sufficient resources, thus enabling real-time capability on edge devices without requiring them to handle the full computational burden
3Device complexity
If the neural network model size is reduced to fit edge devices, then device resource requirements are reduced, but recognition accuracy deteriorates
Solution Approach 1:
The patent segments the recognition task into feature extraction and classification components, with the smaller first neural network handling feature extraction on edge devices and the larger second neural network handling classification on servers. This segmentation allows the model size on edge devices to be reduced to fit resource constraints while the overall system maintains high recognition accuracy through the contribution of the larger second network, thus resolving the contradiction between model size and recognition accuracy
Solution Approach 2:
The first neural network on edge devices acts as an intermediary that extracts essential features from input data and transmits them to the second neural network on servers. This intermediary processing reduces the amount of data that needs to be processed by the larger model, allowing the edge device to use a smaller model while the server-based second network compensates for the reduced accuracy, thus maintaining overall recognition accuracy while reducing model size on constrained devices
Data Source
AI summary
A providing apparatus according to an embodiment of the present disclosure includes a memory and a hardware processor coupled to the memory. The hardware processor is configured to: store, in the memory, a first machine learning model capable of changing an amount of calculation of a model of a neural network; acquire device information; set, based on the device information, extraction conditions representing conditions for extracting second machine learning models from the first machine learning model; extract the second machine learning models from the first machine learning model based on the extraction conditions; and provide the second machine learning models to a device specified by the device information.