DNNFusion Operator Abstraction for Mobile Deep Neural Network Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural network (DNN) execution frameworks face inefficiencies due to restrictive fusion patterns that fail to capture the diversity of operators and layer connections, especially in extremely deep models, leading to high memory and computational requirements, making it challenging to deploy DNNs on resource-constrained mobile devices.
Innovation Solution
The DNNFusion framework classifies operators into high-level abstract types, performs mapping type analysis, and generates optimized fusion code using a novel mathematical property-based graph rewriting approach, enabling more aggressive optimizations and increased fusion opportunities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional pattern-based operator fusion is used, then implementation simplicity is maintained, but fusion opportunities are limited and cannot cover diverse operators and layer connections
Solution Approach 1:
The patent changes the parameter of operator representation from concrete operation types to abstract mapping types (one-to-one, one-to-many, many-to-many). This abstraction enables the fusion framework to handle diverse operators uniformly without increasing complexity, as the mapping type captures the essential data flow pattern regardless of the specific operator
Solution Approach 2:
The patent creates a universal fusion framework that works across all operator types by classifying them into mapping types. A single fusion algorithm can handle any combination of operators as long as their mapping types are known, making the framework multi-functional without requiring separate handling for each operator type
2Measurement precision
If DNN models are made deeper to achieve higher accuracy, then model performance improves, but memory and computational requirements increase
Solution Approach 1:
The patent merges multiple operators into fused operations, eliminating intermediate results and reducing memory access. By combining consecutive operators with compatible mapping types into single fused operators, the framework reduces the total number of operations and intermediate data structures, thereby decreasing memory and computational requirements
Solution Approach 2:
The patent extracts and eliminates unnecessary intermediate materialization steps from the computation graph. By identifying operators that can be fused, the framework removes intermediate result storage and retrieval operations, reducing memory usage without affecting the final computation outcome
3Productivity
If operator fusion is applied to reduce memory access, then execution efficiency improves, but the fusion patterns must be restrictive to ensure correctness
Solution Approach 1:
The patent changes the parameter used for fusion decision-making from specific operator type matching to mapping type compatibility. This parameter change allows the framework to determine fusion feasibility based on data flow patterns rather than concrete operator identities, expanding the range of fusible operator combinations while maintaining correctness guarantees
Data Source
AI summary
This disclosure has presented a new loop fusion framework called DNNFusion. The key advantages of DNNFusion include: 1) a new high-level abstraction comprising mapping type of operators and their combinations and the Extended Computational Graph, and analyses on these abstractions, 2) a novel mathematical-property-based graph rewriting, and 3) an integrated fusion plan generation. DNNFusion is extensively evaluated on 15 diverse DNN models on multiple mobile devices, and evaluation results show that it outperforms four state-of-the-art DNN execution frameworks by up to 8.8× speedup, and for the first time allows many cutting-edge DNN models not supported by prior end-to-end frameworks to execute on mobile devices efficiently (even in real-time). In addition, DNNFusion improves both cache performance and device utilization, enabling execution on devices with more restricted resources. It also reduces performance tuning time during compilation.