Neural Network Resource Allocation and Placement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing neural network processing systems face delays due to the time-consuming process of resource allocation and placement, which reduces system responsiveness and user experience, as the compiler needs to parse the neural network model and determine available resources before generating instructions for processing.
Innovation Solution
The proposed solution decouples resource allocation and placement by allowing the compiler to generate instructions ahead of time, specifying memory and computation resources, while the runtime engine performs dynamic placement upon request, enabling faster processing and improved responsiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the compiler parses the neural network model and determines available resources before generating instructions, then resource allocation accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies preliminary action by having the compiler generate resource allocation information in advance during the compilation phase, before runtime execution. The compiler parses the neural network model and determines resource requirements ahead of time, storing this allocation information for later use. This separates the time-consuming analysis work from the runtime processing, improving both accuracy and speed during actual execution.
2Speed
If the compiler generates instructions specifying memory and computation resources ahead of time, then system responsiveness is improved, but resource placement flexibility is reduced
Solution Approach 1:
The patent segments the resource management process into two independent phases: compilation time (resource allocation) and runtime (resource placement). The compiler phase handles high-level resource allocation and generates instructions, while the runtime engine handles specific resource placement decisions. This segmentation allows pre-computation for speed while maintaining runtime flexibility for adaptability.
Solution Approach 2:
The patent introduces an intermediary layer of resource allocation information that bridges the compiler and runtime engine. This intermediary structure contains resource requirements and allocation strategies generated by the compiler, which the runtime engine then uses to make specific placement decisions. This mediator preserves both the speed benefits of pre-computation and the flexibility of runtime adaptation.
Data Source
AI summary
Techniques for operating a computing system to perform neural network operations are disclosed. In one example, a method comprises receiving a neural network model, determining a sequence of neural network operations based on data dependency in the neural network model, and determining a set of instructions to map the sequence of neural network operations to the processing resources of the neural network processor. The method further comprises determining, based on a set of memory access operations included in the set of instructions, a first set of memory references associated with a first location of an external memory to store the input data and a second set of memory references associated with a second location of the external memory to store the output data, and generating an instruction file including the set of instructions, the first set of memory references and the second set of memory references.