AI Inference Compiler Runtime for Heterogeneous Neural Network Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous vehicles and robots face challenges in efficiently compiling and deploying complex neural network architectures due to increased data volume and resource demands, particularly in remote deployment scenarios, which affect computational efficiency and hardware-software performance.
Innovation Solution
A system and method that involves partitioning neural network architectures into sub-networks, assigning these to heterogeneous processing units, and optimizing execution schedules using a graph partitioner and schedule optimizer to improve computational efficiency and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural network architectures are deployed remotely to autonomous vehicles, then deployment flexibility is improved, but computational efficiency and resource utilization deteriorate
Solution Approach 1:
The neural network architecture is divided into multiple sub-networks that can be independently assigned to different processing units (GPUs, FPGAs, CPUs). This segmentation enables efficient resource utilization by matching specific sub-networks to hardware components with appropriate computational characteristics, thereby maintaining high computational efficiency while supporting remote deployment flexibility.
Solution Approach 2:
The system implements dynamic assignment of sub-networks to processing units based on real-time computational demands and hardware capabilities. The runtime environment can reconfigure which processing unit executes which sub-network, optimizing resource allocation and maintaining computational efficiency across diverse deployment scenarios.
2Power
If complex neural network architectures are used for autonomous navigation, then processing capability is improved, but hardware resource demands and software complexity worsen
Solution Approach 1:
Complex neural network architectures are segmented into smaller sub-networks that can be independently compiled and assigned to specific processing units. This reduces the complexity burden on any single hardware component and simplifies the software compilation process by handling smaller, more manageable units separately.
Solution Approach 2:
A runtime environment acts as an intermediary between the neural network software layers and the heterogeneous hardware processing units. This runtime manages the complex interactions, resource allocation, and communication between components, thereby reducing the complexity burden on both the software architecture and hardware implementation.
3Adaptability or versatility
If multiple compilers are used to generate execution instructions for different processing units, then hardware adaptability is improved, but compilation time and processing overhead worsen
Solution Approach 1:
Multiple compilers are pre-configured and prepared before deployment, each optimized for specific processing unit types (GPU, FPGA, CPU). This preliminary preparation allows for faster compilation during runtime by avoiding the need to develop or configure compilers on-demand, thereby reducing compilation time while maintaining hardware adaptability.
Solution Approach 2:
Instead of creating a single universal compiler that must handle all hardware types, the system uses multiple specialized compilers that are copies or variants of a base compiler framework. Each compiler is optimized for its target hardware architecture, reducing compilation complexity and time while maintaining broad hardware adaptability through the selection of appropriate compiler copies.
Data Source
AI summary
Embodiments include systems and methods for processing sensor data and generating operational instructions of hardware of egos (e.g., autonomous vehicles, robots). The ego includes any number of machine-learning architectures, often neural network architectures, for processing sensor data and recognizing the environment around the ego and making decisions on the ego's behavior. The neural network architectures of the ego ingest sensor data and execute any number of operations related to a particular domain or task, such as object recognition or path planning, using the sensor data. A graph partitioner is trained to assign functions in the software of the neural networks and the sensor data to certain hardware processing units. Several compilers are used to generate the instructions based upon the assigned type of processing unit.