Distributed Inference via Segmented Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems face challenges in performing distributed inference on edge devices wirelessly connected to the cloud, due to dynamic wireless channel conditions, energy constraints, and varying prediction accuracy between local and remote inference approaches.
Innovation Solution
A method and system for distributed inference that processes data samples using multiple deep neural network inference functions on both mobile devices and server devices, allowing for local and remote inference decisions based on energy-saving and accuracy modes, and combining predictions for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If local inference is performed on the mobile device, then response delay is reduced, but prediction accuracy deteriorates
Solution Approach 1:
The DNN is segmented into local layers deployed on the mobile device and remote layers deployed on the cloud server. The mobile device can execute local layers independently for fast local exit, or forward intermediate features to the cloud for remote layers when higher accuracy is needed, thus resolving the contradiction between speed and accuracy through hierarchical processing.
Solution Approach 2:
The system dynamically decides whether to use local exit or remote exit based on real-time conditions such as channel quality, energy availability, and accuracy requirements. This dynamic switching mechanism allows the system to adapt between local inference (fast) and remote inference (accurate) modes, balancing the trade-off between response delay and prediction accuracy.
2Measurement precision
If remote inference is performed on the cloud server, then prediction accuracy is improved, but energy consumption increases
Solution Approach 1:
By segmenting the DNN into local and remote layers, the system enables the mobile device to handle simple tasks locally without cloud communication, saving energy. Only when higher accuracy is required does the system activate remote layers on the cloud, thus reducing overall energy consumption while maintaining accuracy when needed.
Solution Approach 2:
The mobile device performs partial inference using local layers for tasks that don't require full model capacity. This partial action suffices for many cases, avoiding the excessive energy cost of always transmitting data to the cloud. Remote layers are activated only when the partial local processing is insufficient.
3Adaptability or versatility
If the DNN is split into local and remote layers, then flexibility is improved, but system complexity increases
Solution Approach 1:
The mobile device acts as an intermediary that coordinates between local execution and remote cloud processing. It manages the splitting of DNN layers, handles feature forwarding, and integrates results from both local and remote exits. This intermediary role provides flexibility while abstracting the complexity of distributed inference from the user.
Solution Approach 2:
The mobile device is designed with multi-functionality, capable of both executing local DNN layers independently and coordinating with remote cloud layers. This universal design allows the same device to adapt to different inference scenarios (local-only, remote-only, or hybrid) without requiring separate specialized systems, managing complexity through versatile components.
Data Source
AI summary
Disclosed are a method and a system of distributed inference. The method comprises: processing a data sample (x) in accordance with a first deep neural network, DNN, inference function (F0, 21) of a mobile device; initiating a local and/or a remote inference in accordance with a second DNN inference function (FC, 22) of the mobile device; forwarding the processed data sample (z) in accordance with the initiated local and/or remote inference; determining an local exit result (y1) of the initiated local inference in accordance with a third DNN inference function (F1, 24) of the mobile device and the processed data sample (z); determining a remote exit result (y2) of the initiated remote inference in accordance with a fourth DNN inference function (F2, 31) of the server device and the processed data sample (z); and providing a final result (yfinal) of the initiated local and/or remote inference.