Neural Network Partitioning for Multi-Core Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern multi-core devices and compilers face challenges in optimizing neural networks due to inefficient data partitioning and operator management, leading to suboptimal performance and resource utilization.
Innovation Solution
A method and apparatus that convert a neural network into partitions based on hardware information and data division directions, optimizing each partition by removing redundant operators and adjusting layer sizes, and converting back to the original format with added synchronization operators for efficient execution on multi-core devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the neural network is divided into partitions based on hardware information and data division directions, then data partitioning efficiency is improved, but the complexity of operator management increases
Solution Approach 1:
The neural network is divided into multiple partitions based on hardware information and data division directions (height, width, or channel directions). Each partition contains a subset of layers and operators that can be independently executed on different hardware devices, enabling efficient data partitioning while maintaining manageable operator complexity through systematic segmentation.
Solution Approach 2:
A compiler is introduced as an intermediary system that automatically manages the complexity of operator management. The compiler handles operator allocation, data type conversion, and synchronization between partitions, allowing the system to benefit from complex partitioning without manually managing the resulting operational complexity.
2Loss of energy
If redundant operators are removed from partitions, then resource utilization is improved, but the precision of neural network operations may be affected
Solution Approach 1:
The compiler automatically identifies and removes redundant operators from each partition that do not contribute to the computational functionality. This extraction of unnecessary operators optimizes resource utilization by reducing computational overhead while preserving the essential neural network operations and their precision through careful analysis of operator dependencies and data flow.
Solution Approach 2:
The system dynamically adjusts operator parameters such as data types, precision levels, and operation modes based on the specific requirements of each partition and the underlying hardware. This allows the system to optimize resource utilization by using lower precision operations where acceptable while maintaining high precision where critical, thereby balancing resource efficiency with operational accuracy.
3Productivity
If the neural network is converted into partitions and back to the original format, then execution efficiency on multi-core devices is improved, but the time required for conversion increases
Solution Approach 1:
The neural network is converted into partitions during the compilation phase before execution begins. This preliminary conversion allows the system to prepare optimized partition structures in advance, so that when execution starts, the partitions are already ready for efficient multi-core processing. The conversion time is amortized over multiple execution iterations, improving overall execution efficiency.
Solution Approach 2:
The system creates a partitioned representation of the neural network that can be efficiently executed on multi-core devices, and then converts this partitioned version back to the original format for storage or further processing. This copying and conversion process enables the system to leverage parallel computing capabilities while maintaining compatibility with existing frameworks and formats.
4Adaptability or versatility
If data division direction is set to multiple directions, then adaptability to different hardware configurations is improved, but the complexity of partition generation increases
Solution Approach 1:
The partition generation system is designed to handle multiple data division directions (height, width, channel) and can adapt to various hardware configurations through a unified approach. The same partitioning framework can accommodate different hardware topologies and requirements by automatically selecting appropriate division directions and partition strategies, thereby achieving versatility without proportionally increasing complexity.
Solution Approach 2:
The system dynamically determines the optimal data division direction and partition structure based on the specific hardware configuration and neural network characteristics. Rather than using a fixed partitioning approach, the system adapts its partition generation strategy in real-time, selecting from multiple possible directions and configurations to achieve optimal performance for each specific hardware setup.
Data Source
AI summary
A method of processing data is performed by a computing device including processing hardware and storage hardware, the method including: converting, by the processing hardware, a neural network, stored in the storage hardware, from a first neural network format into a second neural network format; obtaining, by the processing hardware, information about hardware configured to perform a neural network operation for the neural network and obtaining partition information; dividing the neural network in the second neural network format into partitions, wherein the dividing is based on the information about the hardware and the partition information, wherein each partition includes a respective layer with an input thereto and an output thereof; optimizing each of the partitions based on a relationship between the input and the output of the corresponding layer; and converting the optimized partitions into the first neural network format.