Distributing Neural Network Inference via State Inference Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed inference systems for artificial neural networks face inefficiencies as they often overburden individual devices with excessive computation, and current methods fail to accurately determine inference distribution ratios considering dynamic device states such as CPU/GPU usage and application execution.
Innovation Solution
An electronic device that includes a memory, a transceiver, and a processor to obtain and process state information from multiple devices, using a state inference model to determine an inference distribution ratio based on CPU, GPU usage, temperature, and application execution, normalizing elapsed time to allocate tasks efficiently among devices, and partitioning the neural network accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If inference is distributed across multiple devices, then computational load on individual devices is reduced, but determining accurate inference distribution ratios becomes complex due to dynamic device states
Solution Approach 1:
The system continuously monitors device state information (CPU usage, GPU usage, temperature, application execution status) and uses this feedback to dynamically adjust inference distribution ratios. The state inference model processes current device states to determine optimal distribution, creating a closed-loop control system that adapts to changing conditions.
Solution Approach 2:
Each device autonomously reports its own state information to the inference distribution determination device, eliminating the need for complex centralized monitoring. Devices self-report CPU usage, GPU usage, temperature, and application execution status, allowing the system to determine distribution ratios based on self-provided data without increasing system complexity.
2Measurement precision
If real-time device state monitoring is implemented, then inference distribution accuracy is improved, but system resource overhead increases
Solution Approach 1:
The system extracts only the essential state information needed for inference distribution decisions (CPU usage, GPU usage, temperature, application execution status) rather than monitoring all possible device parameters. This selective extraction reduces measurement overhead while maintaining sufficient accuracy for determining optimal inference distribution ratios.
Solution Approach 2:
The system implements partial monitoring by focusing on key performance indicators relevant to inference workloads rather than comprehensive device state tracking. This partial action approach achieves adequate measurement precision for inference distribution while minimizing the resource overhead associated with continuous full-state monitoring.
3Productivity
If dynamic inference distribution is implemented, then resource utilization is optimized, but computational overhead for determining distribution ratios increases
Solution Approach 1:
The state inference model is pre-trained offline using historical device state data and inference performance metrics. This preliminary training phase allows the model to learn optimal distribution patterns without incurring computational overhead during real-time operation. During runtime, the pre-trained model quickly processes current device states to determine distribution ratios, minimizing computation time.
Solution Approach 2:
The system replaces complex real-time optimization algorithms with a pre-trained state inference model that uses learned patterns from historical data. This substitution transforms the computationally intensive real-time optimization problem into a faster inference task, maintaining dynamic resource utilization optimization while reducing the computational overhead and time required for distribution determination.
Data Source
AI summary
Provided is an electronic device including a memory storing a state inference model, and at least one instruction; a transceiver; and at least one processor configured to execute the at least one instruction to: obtain, via the transceiver, first state information of each of a plurality of devices at a first time point, obtain second state information of each of the plurality of devices at a second time point that is a preset time interval after the first time point, by inputting the first state information to the state inference model, and determine an inference distribution ratio of the artificial neural network of each of the plurality of devices, based on the second state information of each of the plurality of devices, where the electronic device is determined among the plurality of devices, based on network states of the plurality of devices.