Neural Network Partitioning for Heterogeneous Hardware Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Partitioning deep neural network (DNN) workloads to fully utilize heterogeneous hardware components in mobile platforms is challenging due to different characteristics of each component, such as CPUs, GPUs, and NPUs, leading to underutilization and suboptimal performance.
Innovation Solution
The implementation of a reinforcement learning-based neural architecture search (RL-NASH) model that partitions DNN models across hardware components, leveraging their unique performance characteristics to optimize latency and accuracy, and uses ensemble techniques to combine outputs from specialized ANNs running on CPUs, GPUs, and NPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DNN workloads are partitioned across heterogeneous hardware components, then hardware utilization improves, but system complexity increases
Solution Approach 1:
The patent segments the DNN workload into multiple independent ANN models, each optimized for specific hardware components. Different processing units (CPU, GPU, NPU) execute specialized ANN models with architectures tailored to their computational characteristics, enabling efficient parallel execution while maintaining manageable system complexity through modular design.
Solution Approach 2:
The system employs a unified framework that manages multiple heterogeneous processing units with different capabilities. The architecture allows the same input data to be processed by multiple specialized ANNs across different hardware components, with results combined to achieve both high hardware utilization and accurate predictions.
2Speed
If specialized ANNs are designed for each processor type, then computation speed improves, but model development complexity increases
Solution Approach 1:
Each ANN model is locally optimized for its target processor type with architecture characteristics matched to that processor's strengths. For example, ANNs for GPU execution may have different layer configurations compared to those for NPU execution, allowing each to leverage its hardware's specific computational advantages for faster processing.
3Measurement precision
If multiple specialized ANNs are executed in parallel, then accuracy improves, but energy consumption increases
Solution Approach 1:
The system maintains continuous useful action by executing multiple specialized ANNs in parallel on different hardware components simultaneously. Rather than sequentially processing through single models, the parallel execution of CPU-optimized, GPU-optimized, and NPU-optimized ANNs on their respective processors maximizes computational throughput and accuracy while distributing energy load across the heterogeneous system.
Data Source
AI summary
The present disclosure advantageously provides a heterogenous system, and a method for generating an artificial neural network (ANN) for a heterogenous system. The heterogenous system includes a plurality of processing units coupled to a memory configured to store an input volume. The plurality of processing units includes first and second processing units. The first processing unit includes a first processor and is configured to execute a first ANN, and the second processing unit includes a second processor and is configured to execute a second ANN. The first and second ANNs respectively include an input layer, at least one processor-optimized hidden layer and an output layer. The second ANN hidden layers are different than the first ANN hidden layers.