Edge AI Runtime Partitioning Neural Network Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning applications on edge devices are hindered by the need for either expensive hardware acceleration or downsizing neural networks, which compromises precision and accuracy, especially in real-time and latency-sensitive applications where cloud offloading is not feasible.
Innovation Solution
An optimized AI runtime system that partitions and distributes neural network models across multiple edge devices, allowing for efficient processing without offloading to the cloud, by determining optimal split locations and deploying workloads on suitable edge devices, thus optimizing throughput and meeting service level objectives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If cloud offloading is used to process heavy neural network workloads, then compute capacity is improved, but latency and connectivity reliability deteriorate
Solution Approach 1:
The neural network model is segmented into multiple partitions that can be distributed across different edge devices. Each partition is deployed to a specific edge device based on network conditions and device capabilities, allowing the workload to be divided into manageable segments that reduce latency while maintaining computational capacity.
Solution Approach 2:
The system transitions from centralized cloud processing to a distributed edge computing architecture by adding the network dimension. Multiple edge devices are introduced as computing nodes, creating a two-dimensional distribution across both computational and network domains, which reduces latency by bringing computation closer to data sources.
2Power
If hardware acceleration equipment (GPUs, TPUs) is purchased to run heavy neural networks on edge devices, then processing capability is improved, but cost increases
Solution Approach 1:
The system enables standard edge devices with general-purpose processors to execute neural network workloads through software-based partitioning and distribution. This universal approach allows any edge device with basic computing capabilities to participate in AI processing, eliminating the need for expensive specialized hardware while maintaining processing capability through intelligent workload distribution.
Solution Approach 2:
Instead of purchasing expensive physical hardware acceleration devices, the system creates virtual copies of neural network models that can be distributed and executed on standard processors. The model is partitioned into multiple versions that can be deployed across different edge devices, providing hardware acceleration functionality through software rather than physical hardware.
3Quantity of substance
If neural networks are downsized to run on standard edge devices, then hardware cost is reduced, but precision and accuracy deteriorate
Solution Approach 1:
The full neural network model is segmented into multiple partitions that are distributed across different edge devices. Each partition maintains the full precision and accuracy of the original model, and the collective output of all partitions combines to provide the complete inference capability, eliminating the need to downsize the model while reducing hardware requirements through distribution.
Solution Approach 2:
Multiple edge devices are merged into a distributed computing system that collectively executes the neural network model. The computational results from all edge devices are combined to produce the final inference output, allowing standard hardware to achieve the precision and accuracy of more powerful systems through collaborative processing.
4Ease of operation
If a monolithic pre-trained DNN is deployed on edge devices, then implementation simplicity is improved, but resource consumption and feasibility deteriorate
Solution Approach 1:
The monolithic DNN is segmented into multiple smaller partitions that can be distributed across different edge devices. Each partition is lighter in resource consumption, and the distribution across multiple devices provides collective computational power, reducing the resource burden on any single device while maintaining the full model capability.
Solution Approach 2:
The system dynamically determines optimal partition locations and deployment configurations based on network conditions, device capabilities, and workload characteristics. This dynamic adaptation allows the system to optimize resource consumption in real-time while maintaining implementation simplicity through automated configuration.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
This disclosure describes techniques and mechanisms for enabling a user and third party applications to dynamically partition and place heavy deep learning workloads on standard edge networks to optimize the overall inference throughput of the network while meeting Service Level Objective(s) (SLOs). The techniques may include profiling, partitioning, and splitting of the deep learning workloads, which may be hidden from the user and/or third party application. The user may user interact with a pre-deployed service through a simple SDK that resembles those used for hardware acceleration, such that the current techniques may be easily inserted into their code.