Heterogeneous Scheduler for Neural Network Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices with heterogeneous hardware (CPU, GPU, neural processor) face challenges in efficiently utilizing all processors simultaneously, leading to idle time and suboptimal performance in executing neural network models, particularly in resource-constrained end-user devices.
Innovation Solution
A framework that compiles neural network models into executable code and uses a heterogeneous scheduler to perform weighted traversals of execution graphs, dynamically dispatching operations across CPU, GPU, and neural processors to maximize concurrent runtime and minimize downtime by analyzing processor utilization and dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing electronic devices use heterogeneous hardware (CPU, GPU, neural processor) for neural network execution, then computing functionality is provided, but processor idle time increases and utilization efficiency decreases
Solution Approach 1:
The scheduler dynamically assigns operations to different processors based on real-time availability and task requirements. The system continuously monitors processor status and adjusts operation dispatching to ensure optimal utilization, preventing idle time by adaptively allocating tasks to available hardware components.
Solution Approach 2:
The system maintains continuous productive action by ensuring that whenever a processor becomes available, new operations are immediately dispatched to it. The scheduler keeps all heterogeneous processors continuously engaged in useful computation by managing operation queues and dependency graphs, eliminating gaps and idle periods in processor utilization.
2Adaptability or versatility
If neural network models are executed on heterogeneous hardware, then computing capabilities are enhanced, but device complexity increases
Solution Approach 1:
The scheduler acts as an intermediary layer between the neural network model and the heterogeneous hardware components. It translates high-level model operations into hardware-specific tasks, manages the complexity of assigning operations to different processors, and handles data transfer coordination. This mediator abstracts the hardware complexity from the model execution logic.
Solution Approach 2:
The system segments neural network operations into distinct computational tasks that can be independently assigned to different processor types. By dividing the model execution into separable operations (convolution, pooling, activation, etc.), the scheduler can optimally distribute them across CPU, GPU, and neural processors based on their specific capabilities and current availability.
3Productivity
If operations are dynamically dispatched across multiple processors, then concurrent runtime is maximized, but scheduling complexity increases
Solution Approach 1:
The system performs preliminary analysis of operation dependencies and processor capabilities before scheduling. By pre-computing the dependency graph of neural network operations and analyzing which processors are best suited for each operation type, the scheduler can make informed dispatching decisions that maximize concurrent execution while managing complexity through structured decision-making frameworks.
Data Source
AI summary
The subject technology determines input parameters and an output format of algorithms for a particular functionality provided by an electronic device. The subject technology determines an order of the algorithms for performing the particular functionality based on temporal dependencies of the algorithms, and the input parameters and the output format of the algorithms. The subject technology generates a graph based on the order of the algorithms, the graph comprising a set of nodes corresponding to the algorithms, each node indicating a particular processor of the electronic device for executing an algorithm. Further, the subject technology executes the particular functionality based on performing a traversal of the graph, the traversal comprising a topological traversal of the set of nodes and the traversal being based on a score indicating whether selection of a particular node for execution over another node enables a greater number of processors to be utilized at a time.