Dynamic Inference Routing for Neural Network Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches to distributed deep neural networks (DDNNs) for inference face challenges in training models with multiple exits and making offloading decisions, particularly due to reliance on local prediction confidence, which is limited to classification tasks and does not generalize to regression tasks, and lacks flexibility in determining the optimal location for inference execution.
Innovation Solution
A learnable control function is implemented to decide whether to process data samples locally or remotely based on the input sample and intermediate outputs, exploiting semantic characteristics to determine the best location for inference, allowing for reduced energy consumption and potentially faster and more accurate predictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If local inference is performed on the device, then response delay is reduced, but prediction accuracy deteriorates due to limited model size
Solution Approach 1:
The patent segments the deep neural network into multiple parts: a first set of layers executed locally on the device and a second set of layers executed remotely on the cloud server. This segmentation allows the system to balance between local processing speed and remote processing accuracy by distributing computational tasks across different locations based on their respective capabilities.
2Measurement precision
If remote inference is performed on the cloud, then prediction accuracy is improved through larger models, but response delay increases due to transmission time
Solution Approach 1:
The patent segments the deep neural network into multiple parts: a first set of layers executed locally on the device and a second set of layers executed remotely on the cloud server. This segmentation allows the system to balance between local processing speed and remote processing accuracy by distributing computational tasks across different locations based on their respective capabilities.
Solution Approach 2:
The system performs preliminary processing by executing the first set of layers locally on the device before transmitting intermediate results to the cloud. This preliminary action reduces the amount of data that needs to be transmitted and allows the cloud server to focus computational resources on the most demanding parts of the inference task, thereby reducing overall response delay.
3Use of energy by moving object
If local inference is performed on the device, then energy consumption is reduced, but model capability is limited due to device constraints
Solution Approach 1:
The patent segments the deep neural network into multiple parts: a first set of layers executed locally on the device and a second set of layers executed remotely on the cloud server. This segmentation allows the system to balance between local processing speed and remote processing accuracy by distributing computational tasks across different locations based on their respective capabilities.
Solution Approach 2:
The system implements a hybrid inference architecture that can dynamically adapt to different tasks and data types by routing computations between local and remote resources. This multi-functional approach allows the same device to handle both energy-efficient local tasks and capability-intensive remote tasks, thereby increasing overall system adaptability.
4Adaptability or versatility
If remote inference is performed on the cloud, then model capability is enhanced through larger models, but energy consumption increases due to transmission and processing
Solution Approach 1:
The patent segments the deep neural network into multiple parts: a first set of layers executed locally on the device and a second set of layers executed remotely on the cloud server. This segmentation allows the system to balance between local processing speed and remote processing accuracy by distributing computational tasks across different locations based on their respective capabilities.
Solution Approach 2:
The system performs partial inference locally on the device rather than transmitting all computations to the cloud. By executing the first set of layers locally and only transmitting intermediate results for the second set of layers, the system reduces the total energy consumption associated with data transmission and cloud processing while still leveraging remote model capability when needed.
Data Source
AI summary
A device (400, 703) for processing a data sample (401, 701) to form a predicted output (409, 410) configured to: receive (801) the data sample (401, 701); input (802) the data sample (401, 701) and/or one or more of any intermediate outputs (406) derived from the data sample to a learnable control function (402); and in dependence on an output (403) of the function (402), perform (803) one of the following: (i) process the data sample (401, 701) to form the predicted output (409) using a first model (404, 405, 704) stored locally; and (ii) send the data sample (401, 701) and/or one or more of the any intermediate outputs (406) to a remote location (407, 702) for input to a second model (408, 705) stored remotely to form the predicted output (410).