Split Neural Inference Using Dynamic Edge-Server Layer Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing inference processing systems face challenges in balancing computation efficiency and security, particularly when using convolutional neural networks, as devices with limited computation capacity require extended processing times and transmitting images to a server increases security risks.
Innovation Solution
An inference processing system where a server and edge device cooperate, with the edge device performing initial computations based on detected image content and determining layer positions, and the server completing the processing to ensure security and reduce overall computation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of intermediate layers is increased to improve inference accuracy, then the accuracy of the final inference result is improved, but the processing time is extended due to the huge computational load
Solution Approach 1:
The patent divides the neural network into multiple intermediate layers and assigns different layers to different devices (edge device and server) based on their computational capabilities. This segmentation allows the system to maintain high inference accuracy by utilizing deeper layers while reducing processing time by distributing computation across multiple devices with different performance levels.
2Productivity
If the input image is transmitted to a server to reduce processing time on the edge device, then the processing time is reduced, but the security risk increases due to potential privacy information leakage
Solution Approach 1:
The patent applies local quality by transmitting only specific intermediate layer data (feature maps from certain layers) rather than the entire input image to the server. This selective transmission reduces the amount of sensitive information sent over the network while still enabling the server to perform computations that reduce overall processing time.
Solution Approach 2:
The edge device performs preliminary processing by computing intermediate layers locally before transmitting the results to the server. This preliminary action reduces the computational burden on the server and minimizes the amount of data that needs to be transmitted, thereby reducing security risks while maintaining processing efficiency.
3Loss of time
If the processing is stopped at an intermediate layer of a shallower hierarchical level to reduce computation on the edge device, then the computation time on the edge device is reduced, but the total computation time increases due to unnecessary computation distribution
Solution Approach 1:
The patent implements a dynamic approach where the division of computational tasks between the edge device and server is not fixed but can be adjusted based on the specific characteristics of the input image and the capabilities of the available devices. This allows the system to optimize the total computation time by flexibly assigning intermediate layer processing to the most appropriate device for each situation.
Data Source
AI summary
An inference processing system performs computation of an inference model by a server and an edge device. Setting in the server, a server-side inference model. In the edge device, an edge-side inference model including an input layer and first to third intermediate layers of the inference model is set, and layer position information indicating a position of a last layer to process the input image in the edge-side inference model is generated. When layer data that is a computation result at the edge device obtained by performing processing from the input layer to a layer indicated by the layer position information and the layer position information is transmitted from the edge device to the server, the layer data is input to a layer subsequent to a layer indicated by the layer position information in the server-side inference model, and performs processing up to the output layer.