Compiler Buffer Allocation for Neural Network Tensor Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multipath Convolutional Neural Networks (CNNs), binary tensor operations face challenges with asynchronous operand reception, leading to potential data loss and deadlock situations, especially when dealing with a high number of paths.
Innovation Solution
A compiler design is implemented to manage tensor operations in neural network accelerators. It identifies binary tensor operations and allocates buffer space based on the difference in instance counts of input operands from different paths, ensuring that operations are performed without data loss or deadlock.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If buffer space is not allocated for asynchronous operands in multipath CNNs, then device complexity is reduced, but data loss and deadlock occur
Solution Approach 1:
The compiler performs preliminary analysis of the compute graph to determine the number of instances of each operand before execution. Based on this analysis, buffer space is pre-allocated for asynchronous operands. This preliminary action ensures that when operands arrive asynchronously during execution, the necessary buffer space is already available, preventing data loss and deadlock without requiring complex runtime management.
2Reliability
If buffer space is allocated for all possible instances of asynchronous operands, then data loss is prevented, but memory resource utilization is reduced
Solution Approach 1:
The compiler analyzes the compute graph structure and determines the exact number of instances of each operand that will be needed during execution. Buffer space is allocated only for this specific number of instances, rather than allocating for all possible instances. This preliminary calculation ensures sufficient buffer space to prevent deadlock while optimizing memory resource utilization.
Solution Approach 2:
The system changes the parameter of buffer allocation from a static fixed-size allocation to a dynamically calculated allocation based on the specific compute graph and its execution requirements. By analyzing the graph structure and determining the actual number of operand instances needed, the buffer size is optimized to match the actual requirements, preventing both data loss and excessive memory consumption.
3Reliability
If the compiler analyzes and allocates buffer space based on instance counts, then tensor processing reliability is improved, but compilation time increases
Solution Approach 1:
The compiler performs the analysis of compute graph structure and calculation of operand instance counts during the compilation phase, before execution begins. This preliminary action consolidates the work into compile time rather than runtime, ensuring that buffer space is allocated correctly for asynchronous operand handling without adding delays during the actual tensor processing execution.
Data Source
AI summary
Examples disclosed herein relate to using a compiler for implementing tensor operations in a neural network base computing system. A compiler defines the tensor operations to be implemented. The compiler identifies a binary tensor operation receiving input operands from a first output tensor of a first tensor operation and a second output tensor of a second tensor operation from two different paths of the convolution neural network. For the binary tensor operation, the compiler allocates a buffer space for a first input operand in the binary tensor operation based on a difference between a count of instances of the first output tensor and a count of instances of the second output tensor.