Segmented Neural Network Deployment for Edge Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for deploying and using neural networks in edge devices face challenges due to limited bandwidth, unreliable connectivity, and resource constraints, leading to inefficiencies and suboptimal performance, particularly in applications where edge devices operate at significant distances from centralized sites or experience connectivity failures.
Innovation Solution
Implementing edge-based inference and compute platforms that utilize machine learning accelerators and neural network accelerators, enabling local data processing and inference on resource-constrained devices, and employing transfer learning methods to adapt neural networks to specific edge device environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If neural networks are deployed on edge devices with limited resources, then latency is reduced and data security is improved, but device complexity and power consumption increase
Solution Approach 1:
The patent segments the neural network into multiple components: a first neural network portion deployed on the edge device and a second neural network portion remaining on the centralized site. This segmentation allows the edge device to perform local inference with reduced latency while avoiding the burden of hosting the entire complex model, thus resolving the contradiction between latency reduction and device complexity.
Solution Approach 2:
The patent introduces a new dimension of distributed deployment across multiple locations (edge device and centralized site). Instead of hosting the entire model in one location, the system distributes different portions across spatial dimensions, enabling both low-latency local processing and access to centralized computational resources.
2Reliability
If neural networks are deployed on edge devices, then data security is improved, but transmission bandwidth and power consumption increase
Solution Approach 1:
The patent applies partial action by deploying only the first portion of the neural network on the edge device rather than the complete model. This allows the system to achieve data security benefits of local processing while minimizing power consumption by keeping the more computationally intensive second portion on the centralized site.
3Measurement precision
If neural networks are deployed on edge devices, then accuracy is improved, but resource constraints are exacerbated
Solution Approach 1:
The patent segments the neural network model and its resources across two locations. The edge device hosts the first portion that can operate with limited local resources, while the centralized site maintains the second portion that requires more computational resources. This segmentation enables accurate inference without concentrating all resource demands on the constrained edge device.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
Methods, devices and system for updating a neural network on an edge device that has low-bandwidth uplink capability include a centralized site/device that is configured to train and send the neural network to the edge device. In response, the centralized site/device may receive neural network information from the edge device that includes all or portions of a dataset, output activations, and/or overall inference result that is collected or generated in the edge device. The centralized site/device may use the received neural network information to update all or a part of the trained neural network, generate updated neural network information based on the updated neural network, and send the updated neural network information to the edge device.