Neural Network Computational Graph Layer Fusion Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network computational platforms face inefficiencies due to high data exchange frequency between off-chip memory and on-chip buffers, limited hardware resource utilization, and inflexible network topologies, which hinder real-time performance and adaptability.
Innovation Solution
A computational graph-level optimization method that reduces data exchange through layer decomposition and operation fusion, utilizing rules-based subgraph matching and automatic optimization to reconstruct the graph, thereby improving execution efficiency and adaptability to various network topologies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is frequently exchanged between off-chip memory and on-chip buffers during neural network computation, then computation can proceed with available data, but data exchange frequency increases causing performance degradation
Solution Approach 1:
The patent merges multiple adjacent layer operations into a single fused operation, allowing data to be processed continuously without being written back to and reloaded from off-chip memory. This combines multiple computation steps that previously required separate data exchanges into one continuous operation executed entirely on-chip.
Solution Approach 2:
The patent performs preliminary actions by pre-loading all necessary data for fused operations into on-chip buffers before execution begins, and by pre-organizing computational graphs to identify fusion opportunities. This prepares data and computation paths in advance to minimize runtime data exchange with off-chip memory.
2Ease of manufacture
If computational operations are kept separate for ease of implementation, then programming is simpler, but hardware resource utilization decreases
Solution Approach 1:
The patent creates universal fusion rules that can be applied to various types of neural network layers (convolutional, fully-connected, normalization layers, etc.). These rules provide a multi-functional framework that adapts to different layer types and computational patterns, enabling efficient hardware resource utilization across diverse neural network architectures while maintaining systematic implementation.
3Loss of time
If computational graphs are optimized through layer fusion, then data exchange frequency reduces, but computation complexity within fused layers increases
Solution Approach 1:
The patent segments the computational graph into identifiable layer units with clear interfaces, then applies fusion rules that systematically combine specific layer types. This segmentation approach breaks down the complex optimization problem into manageable segments (different layer types, different fusion patterns) that can be processed independently and systematically.
Solution Approach 2:
The patent changes computational parameters by fusing layers with different operation types into unified computational units. This transforms multiple separate operations with different parameters into a single operation with consolidated parameters, reducing the number of parameter sets that need to be managed while executing the fused computation.
4Ease of manufacture
If fixed computational graph structures are used, then implementation is straightforward, but adaptability to different network topologies is limited
Solution Approach 1:
The patent implements dynamic adaptation by providing a set of fusion rules that can be selectively applied based on the specific computational graph structure and layer types present. The optimization process dynamically identifies applicable fusion opportunities and applies appropriate rules, allowing the same framework to adapt to various network topologies including convolutional networks, recurrent networks, and hybrid architectures.
Data Source
AI summary
The present invention discloses a method to optimize a neural network computational graph. The computational graph is used for performing neural network calculation by a computational platform. The computational platform reads data needed by the calculation from off-chip memory. The method comprises: layers which can be fused are selected at least based on an optimization rule to reduce frequency of data exchange between the computational platform and the off-chip memory, carrying out fusion for at least two adjacent layers in the computational graph according to the selected layer objects. Here, the at least two adjacent layers are at least one of the following: horizontally adjacent layers having the same input of feature maps; and longitudinally adjacent layers in which the calculation results of a feature map of a previous layer are at least part of input for a next layer. The method to optimize a computational graph of the present invention can be automatically carried out based on rules or through isomorphic subgraph matching. Thus, an optimal reconstruction mode for executing the computational graph is found out, execution efficiency of the neural network computational platform is improved.