Neural Network Compilation for Dynamic Processor Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems lack the capability to efficiently determine the optimal hardware for executing deep neural network operations, leading to increased computing bandwidth and power consumption, which can negatively impact device performance.
Innovation Solution
The system reorders operations to minimize tensor lifetime and improves cache usage by annotating each node of a neural network model to indicate the best-suited processor (CPU, GPU, or neural processor) for execution, allowing dynamic task allocation and efficient resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If neural network operations are executed on general-purpose processors (CPU/GPU), then computing flexibility is maintained, but power consumption and energy usage increase
Solution Approach 1:
The patent segments the neural network model into multiple operations, each annotated with processor-specific instructions indicating the optimal processor (CPU, GPU, or neural processor) for execution. This segmentation allows each operation to be assigned to the most energy-efficient suitable processor while maintaining overall system flexibility.
Solution Approach 2:
The patent applies local quality by making different parts of the neural network model have different execution characteristics through annotations. Each node or operation is individually annotated with processor-specific instructions, allowing local optimization of energy consumption based on the specific requirements of each operation.
2Productivity
If operations are reordered to minimize tensor lifetime, then cache usage improves and runtime decreases, but operation complexity increases
Solution Approach 1:
The patent performs preliminary action by reordering operations and adding processor-specific annotations during the compilation phase, before the model is deployed for execution. This advance preparation optimizes cache usage and minimizes tensor lifetime without adding complexity during runtime execution.
Solution Approach 2:
The patent introduces processor-specific annotations as an intermediary layer between the neural network model and the execution hardware. These annotations act as a mediator that guides the runtime system in assigning operations to appropriate processors, simplifying the execution complexity while achieving optimized performance.
3Adaptability or versatility
If dynamic task allocation is implemented across multiple processors, then resource utilization improves, but system complexity increases
Solution Approach 1:
The patent implements dynamics by enabling runtime assignment of operations to different processors based on the processor-specific annotations. The system dynamically selects the appropriate processor for each operation during execution, improving resource utilization while the annotations themselves provide a static guide that limits the complexity of the dynamic decision-making process.
Data Source
AI summary
The subject technology runs a compiled neural network (NN) model on a particular processor with multiple priority queues for executing different processes, the compiled NN model being assigned to a particular priority queue, and the compiled NN model includes context switch instructions that were previously inserted into a neural network (NN) model from which the compiled NN model was compiled. The subject technology determines that a particular context switch instruction has been executed by the particular processor. The subject technology determines that a different process is waiting to be executed, the different process being assigned to a different priority queue and the different process being a higher priority process than the running compiled NN model. In response to executing the particular context switch instruction, the subject technology performs a context switch to the different process assigned to the different priority queue when the different process is waiting to be executed.