Heterogeneous AI Processor Allocation for Mixed-Bit Neural Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The selection of hardware for processing neural network models is not clear, leading to suboptimal performance due to varying constraints in performance, processing capability, and power consumption among different hardware types.
Innovation Solution
A system comprising heterogeneous processors is used, where operations for neural network layers with varying bit-width parameters are assigned to the most suitable processors, allowing optimal performance without retraining the neural network model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single type of hardware processor is used to process neural network operations, then device complexity is reduced, but processing capability and performance are limited due to varying bit-width requirements across different layers
Solution Approach 1:
The system divides the neural network processing into segments, with different processor types (NPU, GPU, CPU) handling different layers or portions of the network based on their capabilities. Each processor type is optimized for specific bit-width requirements, allowing the system to process multi-bit-width parameters efficiently without requiring a single complex processor to handle all cases.
Solution Approach 2:
The system creates a universal processing architecture that can handle multiple types of neural network operations across different processor types. The controller dynamically assigns operations to appropriate processors based on their capabilities, making the system adaptable to various bit-width requirements while maintaining a unified interface for neural network processing.
2Productivity
If hardware is selected based on highest processing capability, then productivity is improved, but power consumption increases due to using high-performance processors for all operations
Solution Approach 1:
The system applies local quality by matching specific processor types to specific processing tasks based on their capabilities and power characteristics. High-performance NPUs handle computationally intensive layers requiring fast processing, while lower-power processors handle less demanding operations. This localized optimization ensures that power consumption is proportional to the actual processing requirements of each layer.
Solution Approach 2:
The controller dynamically assigns neural network layers to appropriate processors based on real-time considerations of processing requirements and power constraints. This dynamic allocation allows the system to optimize the balance between processing speed and power consumption by selecting the most appropriate processor for each operation rather than using a fixed high-performance configuration for all tasks.
3Adaptability or versatility
If neural network model is retrained for optimal hardware performance, then processing capability is optimized, but time and cost increase significantly
Solution Approach 1:
Instead of retraining the neural network model for different hardware configurations, the system changes the parameter of processor assignment dynamically. The controller adjusts which processor handles which layer based on the neural network's layer characteristics and processor capabilities, achieving hardware optimization without modifying the model weights or requiring retraining.
Data Source
AI summary
According to an example of the present disclosure, a system is provided. The system may include: a plurality of heterogeneous processors configured to perform inference using a neural network model comprising a plurality of layers; a memory for storing the neural network model, wherein each layer of the neural network model comprises a plurality of parameters having different bit-widths; and a controller for assigning an operation for an arbitrary layer of the neural network model to an arbitrary processor among the plurality of heterogeneous processors. An operation for a first portion of the arbitrary layer of the neural network model may be assigned to a first processor among the plurality of heterogeneous processors. An operation for a second portion of the arbitrary layer of the neural network model may be assigned to a second processor among the plurality of heterogeneous processors. A first bit-width of a first parameter for the first portion and a second bit-width of a second parameter for the second portion may be different from each other.