Edge Inference System Using Neural Network Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inference processing systems face increased computational load when using multiple neural networks, especially in devices with low computational power, and existing solutions do not effectively manage load distribution while maintaining data privacy.
Innovation Solution
The system employs a first terminal to execute processing operations up to a predetermined intermediate layer of a first neural network and transfers intermediate data to a second terminal, where further processing is performed by a second neural network, commonizing processing layers to reduce load and ensure data privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If inference processing is performed using multiple neural networks that output different inference results based on the same input data, then the accuracy and reliability of inference results are improved, but the computational load on the edge device increases significantly
Solution Approach 1:
The patent divides the neural network processing into multiple segments: the edge device executes processing up to a predetermined intermediate layer, while the server executes processing from that intermediate layer onwards. This segmentation allows the edge device to perform only partial processing, reducing its computational load while still contributing to the overall inference result through the intermediate data it generates
Solution Approach 2:
The patent introduces intermediate data as a mediator between the edge device and server. The edge device processes input data to generate intermediate data, which is then transmitted to the server for further processing. This intermediary mechanism enables distributed processing across multiple devices, reducing the burden on any single device while maintaining the ability to produce reliable inference results
2Loss of information
If processing operations up to a predetermined intermediate layer are executed in each neural network on the edge device, then data privacy is maintained, but the computational load increases when using multiple neural networks
Solution Approach 1:
The patent segments the neural network execution such that the edge device performs processing only up to a predetermined intermediate layer, rather than executing complete neural networks. This segmentation reduces the computational load on the edge device while still maintaining data privacy, as the processed intermediate data is transmitted to the server for further processing instead of transmitting raw input data
3Measurement precision
If the number of intermediate layers in the neural network is increased to perform higher dimensional extraction of feature amount, then the accuracy of inference is improved, but the processing time and computational load increase on low computational power devices
Solution Approach 1:
The patent segments the multi-layer neural network processing between the edge device and server, with the edge device executing processing up to a predetermined intermediate layer. This allows the system to utilize deeper neural networks with more intermediate layers for high-dimensional feature extraction without requiring the edge device to complete the entire processing chain, thus maintaining feature extraction accuracy while reducing local processing time
Solution Approach 2:
The patent transitions the computational workload from a single-dimension local processing model to a multi-dimension distributed processing model. By moving the completion of neural network processing to the server dimension, the system can accommodate deeper networks with more intermediate layers without proportionally increasing the processing time burden on the edge device
Data Source
AI summary
An inference processing system that includes a first terminal and a second terminal and performs inference processing using a plurality of neural networks. An image capturing apparatus as the first terminal executes inference processing by a first neural network using acquired data as an input thereto and outputs intermediate data to a server as the second terminal. The intermediate data is obtained by executing processing operations in intermediate layers, up to a predetermined intermediate layer, of the first neural network, which are commonized with a second neural network. The server executes processing operations in intermediate layers, after the predetermined intermediate layer, of the second neural network using the intermediate data as an input thereto.