Hardware Operator Fusion for Neural Network Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The accumulation of memory access latency in hardware accelerators processing artificial neural networks becomes a significant bottleneck as the number of layers/operators increases, necessitating efficient techniques to reduce this latency.
Innovation Solution
The method involves operator fusion, where multiple operators are combined into a single macro operator, with data transfer occurring through a software or hardware-managed on-chip memory, reducing memory access latency and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple operators are executed sequentially with separate memory accesses, then each operator can be implemented independently, but memory access latency accumulates and becomes a bottleneck
Solution Approach 1:
The patent applies operator fusion to combine multiple sequential operators into a single fused operator that processes data through multiple computational steps without intermediate memory accesses. This merging eliminates the accumulation of memory access latency while maintaining independent operator implementation through a unified computational graph representation.
Solution Approach 2:
The fused operator maintains continuous computation by keeping intermediate results in on-chip memory rather than writing to off-chip memory between operators. This continuous action eliminates idle waiting time between operators and ensures that computational operations proceed without interruption, directly reducing the time loss associated with sequential memory accesses.
2Productivity
If operator fusion is implemented to reduce memory access latency, then processing performance improves, but compiler complexity and engineering effort increase
Solution Approach 1:
The patent introduces a universal operator fusion framework that can handle multiple neural network topologies and operator combinations through a single unified approach. The computational graph representation and fusion algorithms work generally across different network architectures, eliminating the need for separate compiler implementations for each topology and reducing overall engineering effort.
Solution Approach 2:
The system performs operator fusion analysis and optimization decisions during the compilation phase before actual execution. By pre-determining the optimal fusion strategy and generating fused operator implementations in advance, the compiler avoids complex runtime decision-making and reduces the perceived complexity during system operation.
Data Source
AI summary
A method of generating executable instructions for a computing system is provided. The method comprises: receiving a first set of instructions including a kernel of a first operator and a kernel of a second operator, the kernel of the first operator including instructions of the first operator and write instructions to a virtual data node, the kernel of the second operator including instructions of the second operator and read instructions to the virtual data node; determining, based on a mapping between the write instructions and read instructions, instructions of data transfer operations between the first operator and the second operator; and generating a second set of instructions representing a fused operator of the first operator and the second operator, the second set of instructions including the instructions of the first operator, the instructions of the second operator, and the instructions of the data transfer operations.